JP2008306264A - Image processor, image processing method, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent images captured in respective conference rooms from mutually be fed back to be displayed one over another on displays etc., in the respective conference rooms. <P>SOLUTION: A first image captured by photographing a member B in a first conference room by a camera in the conference room is displayed in a second conference room. When an image captured by photographing a member A before the displayed first image in the second conference room by a camera therein is displayed as it is in the first conference room, a second image 80 obtained by feeding the first image back is displayed together with the member A. For the purpose, an area 90, shown by a dotted line matching the first image displayed in a photographic area, in the photographed second image is determined. The member A is not included in the area 90. An alpha channel having the area 90 set as a transmission area is constituted and added to the second image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, a program, and a recording medium used for conducting a video conference and the like.

Video conferencing systems have been developed in the past. By using the video conferencing system, video / audio can be transmitted and received between two or more remote conference rooms. We were able to hold a meeting.
On the other hand, as a conventional image processing technology, there is a technology for generating a closed contour without a moving region in a moving image for which a background image is not prepared, and separating the moving region and the background region using the closed contour. It has been developed (see, for example, Patent Document 1).
Japanese Patent Laid-Open No. 11-353486 (first page, FIG. 1 etc.)

  However, in the conventional video conference system, depending on the installation location and orientation of the camera that captures the state of the conference in the first conference room, the camera is not only a participant in the first conference room, There may be a case where an image obtained by photographing the state of the second conference room displayed on a display or the like disposed inside, an image of a document, or the like is photographed. Thus, an image obtained by re-taking an image once displayed on a display or the like usually has poor image quality and is difficult to grasp. Therefore, when such an image is displayed in the second conference room, it is annoying and the image displayed in the display in this image is displayed in the conference room in the second conference room. There has been a problem that attention has shifted and it is not possible to concentrate on the meeting, and there is a possibility that an efficient meeting cannot be realized.

In particular, when an image taken in the first meeting room includes an image including a display that displays the state of the second meeting room, the image taken in the first meeting room is When sent to the second conference room at a remote location and displayed on a display or the like, the display or the like of the second conference room contains images or the like taken in the second conference room that are not originally required to be displayed. Feedback will be displayed. As a result, in the second meeting room, the state of the first meeting room and the state of the second meeting room are displayed on the same screen, and the entire displayed image is congested, and the concentration in the meeting There was a problem that it was impossible.
Specifically, one of the participants speaking in the second meeting room is also displayed on the display of the second meeting room. As a result, in the second meeting room, where and who There was a problem that it was difficult to instantly determine whether or not he was speaking, and the participants could not grasp the situation and were confused and unable to concentrate on the meeting.

  The present invention has been made to solve the above-mentioned problems, and the object thereof is to feed back the first images taken in each conference room as described above and display them on a display or the like in each conference room. Another object of the present invention is to provide an image processing device, an image processing method, a program, and a recording medium that prevent the second image from being displayed on the second image.

In order to solve the above problems, the present invention provides a reception unit that receives a first image, a display unit that displays the first image received by the reception unit, and a first image that the display unit displays. An image receiving unit that receives a second image that is an image of an area that includes the image, and correction for color matching and alignment between the first image and the first image included in the second image Based on the correction parameter, the correction parameter storage unit that stores the correction parameter used in the correction, the first image or the image in the region where the first image in the second image is located is corrected. A comparison unit for comparing the first image with an image in a region where the first image in the second image is located, and using a comparison result of the comparison unit, Set the area that matches the first image of the And transmitting image constructing unit for generating a transmission image, an image processing apparatus characterized by comprising an output section for outputting the transmission image the transparent image forming section is generated.
With this configuration, the color and position are corrected, and it is possible to accurately determine whether the first image in the second image matches the original first image to be photographed. In addition, it is possible to accurately designate and output a region that coincides with the first image displayed in the photographing region of the photographed second image as the transmissive region.

According to the present invention, in the above-described invention, the comparison unit is configured to display, based on the correction parameter, an image in a region where the first image and the first image in the second image are located. It is characterized by correcting.
With such a configuration, the first image in the second image is corrected by correcting both the color and position of the image in the region where the first image in the first image and the second image are located, It is possible to judge with higher accuracy whether the original first image that was the subject of photography matches, and it matches the first image displayed in the photographing area of the second image taken. The area to be processed can be designated and output as a transparent area with high accuracy.

Further, the present invention is the invention described in the above, wherein the transparent image constructing unit configures a plurality of transparent images obtained by cutting out images of regions separated by the transparent regions in the second image. And
With such a configuration, it is possible to delete an image in an area where display is unnecessary and reduce the data amount of the second image.

Further, the present invention is characterized in that, in the above-described invention, the transparent image constructing unit generates a transparent image having an alpha channel for designating a transparent region.
With this configuration, it is possible to display, as a transmissive area, an area that matches the first image displayed in the shooting area of the second image that has been shot without reducing the information of the second image. It becomes.

  Further, according to the present invention, in the above-described invention, the mark image which is an image serving as a mark is a plurality of pattern images arranged at different positions, and the appearance pattern of the mark image is between the pattern images. An image storage unit that stores a plurality of different pattern images, a landmark image detection unit that detects the landmark image from the pattern captured image, and each landmark image detected by the landmark image detection unit, An appearance pattern for the landmark image is acquired, and the position of the landmark image detected in each pattern image and the placement of the landmark image in the plurality of pattern images having the appearance pattern matching the appearance pattern of the landmark image are arranged. A correspondence information acquisition unit that acquires correspondence information indicating a correspondence with a position, and the correction using the correspondence information acquired by the correspondence information acquisition unit And a correction parameter configuration unit that accumulates in the correction parameter storage unit, wherein the display unit switches and displays a plurality of pattern images stored in the image storage unit, and receives the image The unit accepts a pattern photographed image obtained by photographing an image including a plurality of pattern images displayed on the display unit, and the landmark image detection unit is configured to receive the pattern photographed image received from the pattern acceptance image received from the image accepting unit. A mark image is detected.

  With this configuration, it is possible to accurately take the correspondence between the position of the image displayed on the display unit and the image displayed on the display unit in the image captured by the photographing unit. Thus, a highly accurate correction parameter can be obtained. In particular, since the mark image is identified and detected from within the photographed image based on the appearance pattern of the mark image, only a part of the display part can be photographed depending on the photographing condition of the photographing part, and within the display area of the display part. Even if you do not know which part of the image was taken, or when the direction of the image taken on the display unit cannot be determined, the position where the landmark image is placed can be accurately identified from the captured image. Can be accurately taken, and a highly accurate correction parameter can be obtained.

  According to the present invention, in the above-described invention, the image storage unit includes a plurality of the mark images, which are images serving as marks, having any one of a plurality of colors at different positions. The positions of the plurality of mark images are the same among the plurality of pattern images, and the appearance patterns of the colors of the mark images arranged at different positions are different from each other. The plurality of pattern images are stored, and the correspondence information acquisition unit uses each mark image detected by the mark image detection unit, and uses the color of the mark image detected at the same position in each pattern photographed image. The plurality of pattern images having a position in the captured image from which the appearance pattern of the color is acquired and an appearance pattern of a color that matches the appearance pattern of the color And acquiring the correspondence information indicating a correspondence between a position disposed landmarks image in.

  With this configuration, it is possible to accurately take the correspondence between the position of the image displayed on the display unit and the image displayed on the display unit in the image captured by the photographing unit. Thus, a highly accurate correction parameter can be obtained. In particular, since the mark image is identified and detected from within the photographed image based on the appearance pattern of the mark image, only a part of the display part can be photographed depending on the photographing condition of the photographing part, and within the display area of the display part. Even if you do not know which part of the image was taken or if the direction of the image taken on the display unit cannot be determined, the position where the landmark image is placed can be accurately identified from the captured image. The correspondence of the position can be taken accurately, and a highly accurate correction parameter can be obtained.

In the invention described above, the present invention relates to correspondence management, which is information indicating a correspondence between a position where a plurality of landmark images are arranged in the plurality of pattern images and a color appearance pattern at the arranged position. A correspondence management information storage unit for storing information; and the correspondence information acquisition unit acquires the correspondence information from the appearance pattern of the color acquired for the landmark image in the photographed image using the correspondence management information. It is characterized by doing.
With such a configuration, it is possible to accurately take the correspondence between the position of the image displayed on the display unit and the image displayed on the display unit in the image captured by the imaging unit.

In the present invention described above, the image storage unit may be configured such that the number of landmark images in each pattern image is A (A is an integer of 2 or more), and the number of pattern images is B (B is 2 or more). A plurality of pattern images satisfying the relationship of 2 ^B ≧ A are stored.
With this configuration, the number of landmark images can be optimized.

In the present invention described above, the mark images of the plurality of pattern images are different from one another for each pattern image, and the mark image detection unit obtains a mark image detected from each pattern image. The correspondence information acquisition unit acquires the landmark image according to the order in which the pattern images acquired by the landmark image detection unit are arranged, and acquires the correspondence information.
With this configuration, when this captured image captures a plurality of pattern images, it is not necessary to be able to identify the plurality of pattern images, and it is not necessary to consider the order of capturing the pattern images, and the processing is simplified. Can be

Further, according to the present invention, in the invention described above, the correction parameter configuration unit outputs color information that is information about a color of a target image in a pattern captured image received by the image receiving unit, and is output by the output unit. The color information acquisition unit that acquires color information of the target image in the pattern image corresponding to the target image indicated by the correspondence information, and the image reception unit uses the color information acquired by the color information acquisition unit. It further comprises color correction information acquisition means for acquiring color correction information, which is a correction parameter used for correction for color matching of the received image, and accumulating it in the correction parameter storage unit.
With this configuration, color correction information can be acquired using colors at corresponding positions between images, and color correction can be performed accurately by performing color correction using this color correction information. .

According to the present invention, in the invention described above, the correction parameter configuration unit performs correction for image alignment received by the image receiving unit using the position information acquired by the correspondence information acquiring unit. It further comprises position correction information acquisition means for acquiring position correction information which is a correction parameter to be used and storing it in the correction parameter storage unit.
With this configuration, the position can be accurately corrected by correcting the position using this correction information.

  The present invention includes a reception unit that receives a first image, a display unit that displays an image, an image reception unit that receives a second image, the first image, and the second image. Correction parameter storage unit for storing correction parameters used for correction for color matching and alignment with the first image, a comparison unit for comparing images, and a transmission image configuration unit for generating a transmission image And an image processing method in an image processing apparatus comprising an output unit, wherein the reception unit receives the first image, and the display unit displays the received first image. The image receiving unit receives a second image that is an image of a region including the displayed first image, and the comparison unit is based on a correction parameter stored in the correction parameter storage unit. Correcting the first image or the image in the region where the first image is located in the second image; and the comparison unit includes the first image and the second image Comparing the image in the region in which the first image is located, and the transmission image constructing unit uses the comparison result to determine a region in the second image that matches the first image. An image processing method comprising: generating a transparent image set in a transparent region; and outputting the generated transparent image by the output unit.

  In the image processing method invention described above, the image processing apparatus may further include a plurality of pattern images in which landmark images, which are images serving as landmarks, are arranged at different positions. An image storage unit that stores a plurality of pattern images with different appearance patterns of the mark image between the pattern images, a mark image detection unit that detects the mark image from the pattern photographed image, a correspondence information acquisition unit, A correction parameter configuration unit that generates correction parameters to be accumulated in the correction parameter storage unit, and wherein the display unit switches and displays a plurality of pattern images stored in the image storage unit, A step in which an image receiving unit receives a pattern-captured image, which is an image obtained by capturing an image including a plurality of pattern images displayed by the display unit; The mark image detecting unit detects the mark image from the pattern photographed image received by the image receiving unit, and the correspondence information acquiring unit uses each mark image detected by the mark image detecting unit. The appearance image for each of the landmark images is acquired, and the landmark image in the plurality of pattern images having the appearance pattern that coincides with the position of the landmark image detected in each of the pattern photographed images and the appearance pattern of the landmark image. A step of acquiring correspondence information indicating a correspondence with a position where the correction information is arranged, and the correction parameter configuration unit configures the correction parameter using the correspondence information acquired by the correspondence information acquisition unit, and stores the correction parameter. And a step of accumulating in the unit.

  Further, the present invention is the image processing method invention described above, wherein the landmark image that is a landmark image having any one of a plurality of colors is different in the image storage unit. A plurality of pattern images arranged at positions, and the positions of the plurality of mark images are the same between the pattern images, and the color of the mark images arranged at different positions is the same. The plurality of pattern images having different appearance patterns are stored, and the correspondence information acquisition unit is detected at the same position in each pattern photographed image using each landmark image detected by the landmark image detection unit. In addition, a color appearance pattern for the mark image is acquired, and a position in the captured image from which the color appearance pattern is acquired and a color appearance pattern that matches the color appearance pattern are included. Acquiring the correspondence information indicating a correspondence between a position disposed landmark image in the plurality of pattern images that, and further comprising a.

  In addition, the present invention includes a correction parameter storage unit that stores correction parameters used for color matching between the first image and the first image included in the second image and correction for positioning. Receiving the first image on the computer, displaying the received first image, receiving a second image that is an image of an area including the displayed first image, Correcting the image in the region where the first image or the first image in the second image is located based on the correction parameter stored in the correction parameter storage; and Comparing one image with an image in a region where the first image in the second image is located, and using the comparison result, the first image in the second image Matching territory Generating a transmission image set in the transmissive region, and a program for and a step of outputting the generated transmission image.

  Further, the present invention is the above-described program invention, wherein the image processing device further includes a plurality of pattern images arranged at different positions of the mark images that are the images serving as the marks. An image storage unit for storing a plurality of pattern images having different appearance patterns of the landmark image, and a step of switching and displaying the plurality of pattern images stored in the image storage unit; A step of receiving a pattern photographed image that is an image obtained by photographing an image including a plurality of pattern images, a step of detecting the landmark image from within the received pattern photographed image, and using the detected landmark images, The appearance pattern for the landmark image is acquired, the position of the landmark image detected in each pattern photographed image, and the landmark image Obtaining correspondence information indicating correspondence with positions where mark images are arranged in the plurality of pattern images having an appearance pattern that coincides with the appearance pattern, and configuring the correction parameter using the obtained correspondence information. And storing the correction parameter in the correction parameter storage unit.

  Further, according to the present invention, in the invention of the program described above, the mark storage image, which is a mark image having any one of a plurality of colors, is placed in a different position in the image storage unit. A plurality of the plurality of pattern images arranged, and the positions where the plurality of mark images are arranged are the same between the plurality of pattern images, and the appearance patterns of the colors of the mark images arranged at different positions The plurality of pattern images different from each other are stored, and using each detected mark image, a color appearance pattern for the mark image detected at the same position in each pattern image is obtained, The position of the mark image in the plurality of pattern images having the position in the captured image from which the color appearance pattern was acquired and the color appearance pattern that matches the color appearance pattern. The step of acquiring the correspondence information indicating a correspondence between a position that is location further characterized by causing a computer to execute.

  Further, the present invention is a computer-readable recording medium on which the program described above is recorded.

  According to the present invention, it is possible to set and output a region that matches the first image displayed in the photographing region of the photographed second image as the transmissive region.

Hereinafter, embodiments of an image processing apparatus and the like will be described with reference to the drawings. In addition, the component which attached | subjected the same code | symbol in drawing shall perform the same operation | movement.
(Embodiment 1)
FIG. 1 is a block diagram illustrating a specific example of an image processing apparatus with a color correction / position correction function according to the present embodiment.
The image processing apparatus 100 includes a reception unit 101, a display unit 102, an imaging unit 103, an image reception unit 104, a delay buffer 105, a correction parameter storage unit 106, a comparison unit 107, a transmission image configuration unit 108, and an output unit 109.

  The accepting unit 101 accepts input of the first image. In order to simplify the description, image information that is information indicating an image is also referred to as an image as appropriate. The reception here may be, for example, a wired or wireless communication line (for example, the Internet, an intranet, a LAN (Local Area Network), or the like, or an IEEE 1394 cable, a dedicated cable for transmitting video information, or the like. The first information transmitted via a predetermined recording medium (for example, an optical disk, a magnetic disk, a semiconductor memory, etc.), Also good.

  The first image is usually a moving image, but may be a still image. The first image is preferably a color image, but may be a grayscale image. This first image is, for example, an image of a meeting scene. A specific example is an image taken in another conference room and transmitted in real time in a video conference system using the image processing apparatus 100 as a terminal. Further, the first image may be an image obtained by accumulating images obtained by capturing a meeting scene. Further, the first image is not limited to an image obtained by photographing a meeting scene, and may be any image. For example, it may be an image of a material used for a meeting or an image for presentation.

  The first image may be an analog image or a digital image. In the case of an analog image, the first image may be a composite signal or a component signal, for example. In the case of a digital image, the first image is, for example, MPEG (Moving Picture Experts Group) or ITU-T H.264. Any algorithm such as H.264 or the like may be used. An uncompressed image may be output. In addition, the image may be of any quality, resolution, etc., such as an HD image. Further, the first image may include or be added with synchronization information for synchronizing with other images and the like.

  The receiving unit 101 may temporarily store the received first image in a delay buffer 105 described later, a memory (not shown), or the like. The receiving unit 101 may receive audio information corresponding to the first image in addition to the first image. The receiving unit 101 may or may not include a device for receiving (for example, a modem or a network card). The accepting unit 101 may be realized by hardware, or may be realized by software such as a driver that drives a predetermined device.

  When the first image received by the receiving unit 101 is an analog signal, an AD (analog-digital) conversion unit (not shown) that converts the first image into a digital signal as appropriate and outputs it in the image processing apparatus 100. Etc. are preferably provided. By providing such an AD conversion unit and the like, it is possible to appropriately digitize the processing performed on the first image. For example, the analog first image received by the reception unit 101 may be converted into a digital signal by such an AD conversion unit and stored in the delay buffer 105.

  The display unit 102 displays the first image received by the receiving unit 101. Specifically, the display unit 102 configures display information using the first image output from the receiving unit 101, and displays a display device such as a liquid crystal display, a plasma display, or a CRT (Cathode Ray Tube) display. Display the first image. Note that the display unit 102 may output the sound provided corresponding to the first image together with the first image. The resolution at which the display unit 102 displays the first image does not matter. In addition, when the first image includes transmission information such as mask information, the display unit 102 transmits the partial image of the first image using the information, or displays the first image. A part of the area may be replaced with another image, for example, a background image or an image of a specific color and displayed. The display unit 102 may or may not include a display device and a speaker that perform display. The display unit 102 may be realized by hardware, or may be realized by software such as a driver that drives a display device.

  The imaging unit 103 captures a second image that is an image of an area including the first image that the display unit 102 displays using a display device or the like. Specifically, the imaging unit 103 captures an area including at least a part of the area on which the display unit 102 displays the first image as an imaging area that is an imaging target area, and the second image To get. More specifically, the imaging unit 103 captures an image of an area including a display surface of a display device such as a display that displays the first image as the second image. The second image is, for example, an image of a video conference in a state in which a participant of the conference is located before the display device of the display unit 102 or the like. The image photographed by the photographing unit 103 is usually a moving image, but may be one or more still images. Note that a plurality of still images taken at predetermined time intervals that are constant or indefinite are moving images.

  The second image may be an analog signal or a digital signal. The second image is usually a color image, but may be a grayscale image. The angle formed by the photographing direction of the photographing unit 103 and the display surface of the first image displayed on the display unit 102 is not limited. The imaging unit 103 is a camera that captures an image, for example, and images the image using an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). Further, when the photographing unit 103 also acquires audio information, the photographing unit 103 may include a microphone, for example. There is no limitation on the data structure, timing, or the like for the imaging unit 103 to output an image. For example, the photographing unit 103 may output the acquired moving image information as one moving image file. The photographing unit 103 may sequentially output image information of each frame constituting the moving image or image information corresponding to each frame every time it is acquired, or when a predetermined number of frames are acquired. Thus, the image information may be output in units of a predetermined number of frames.

  In addition, the photographing unit 103 may create a moving image in which time position information that is information that can specify an arbitrary temporal position in the moving image is added to the captured moving image. For example, a moving image to which information indicating a time position such as information indicating the time at the time of shooting is added may be created as the time position information. For example, the image capturing unit 103 adds, to the image information of each frame constituting the moving image, information on the time at which the image information was acquired as time position information, specifically, a time code or image information added as a time stamp. May be created. Further, frame identification information for identifying a frame may be added to each frame of the image information. The image information created and output by the photographing unit 103 may be analog image information or digital image information. For example, in the case of digital image information, MPEG or ITU-T H.264 is used. H.264 and other algorithms for compressing and encoding image information may be used. Further, uncompressed image information may be output. The image information may be any image quality, resolution, etc. such as an HD image. Further, the photographing unit 103 may include a microphone and acquire voice information.

  The imaging unit 103 may output device identification information for identifying the imaging unit 103. When the photographing unit 103 is a photographing device such as a camera, the device identification information may be a manufacturing number of the photographing device or an identification number set in the photographing unit 103 in advance. . Further, it may be physical address information on the network such as the IP address of the photographing unit 103. The device identification information may be information that can identify the imaging unit 103 as a result, and may be information that identifies an input terminal such as a port number to which the imaging unit 103 is connected. The device identification information is stored in advance in a memory (not shown) in the photographing unit 103, for example. Moreover, it does not ask how the imaging | photography part 103 transmits identification information. Specifically, the identification information can be output by the same method as the position identification information described above.

  The image receiving unit 104 receives a second image that is an image of an area including the first image displayed on the display unit 102. Here, specifically, the second image output by the imaging unit 103 and captured by the imaging unit 103 is received. This reception may be, for example, a wired or wireless communication line (for example, the Internet, an intranet, a LAN, or the like, or an IEEE 1394 cable or a dedicated cable for transmitting video information). Information received via a predetermined recording medium (for example, an optical disk, a magnetic disk, a semiconductor memory, etc.) may be received. In the present embodiment, an example will be described in which the image receiving unit 104 receives information indicating a second image captured by the capturing unit 103 via a dedicated cable or the like. The image receiving unit 104 may or may not include a device (for example, a modem or a network card) for receiving. The image receiving unit 104 may be realized by hardware, or may be realized by software such as a driver that drives a predetermined device.

  The delay buffer 105 is for delaying the first image displayed on the display unit 102 by a predetermined time and inputting the first image to the comparison unit 107. The delay buffer 105 is temporarily stored, and is read by the comparison unit 107 after a preset time delay. For example, the first image received by the receiving unit 101 is input to the display unit 102 and simultaneously stored in the delay buffer 105. Specifically, the predetermined delay time of the delay buffer 105 is substantially the same as the time until the second image obtained by capturing the first image read by the display unit 102 is input to the comparison unit 107. , Preferably set to be the same time. For example, a ring buffer or the like can be used as the delay buffer 105. The accumulation of the first image in the delay buffer 105 is performed by a control unit (not shown) or the like.

  The correction parameter storage unit 106 stores correction parameters used for correction for color matching and alignment between the first image and the first image included in the second image. The correction parameters used for correction for color matching are, for example, at least one of the first image and the second image for matching the colors of the first image and the second image. This is color correction information which is information used for color correction. Specifically, the color correction information is a value such as a coefficient of a function used for color correction. The function for correcting the color may be any function. Specific examples of functions will be described later. The color correction described here is, for example, for matching the color of the second image obtained by photographing by the photographing unit 103 with the original color of the first image to be photographed by the photographing unit 103. It is a correction. Note that the color correction in this case may be correction of relative color between the second image and the original of the first image that is the photographing target of the second image, for example, the second image. May be corrected to match the color of the original first image. The color matching described here does not have to aim at perfect matching, but may be any as long as it is easy to determine a matching area between images as described later.

  The correction parameter used for correction for alignment is, for example, at least one of the first image and the second image for aligning corresponding positions of the first image and the second image This is position correction information used for position correction of the image. The position correction includes correction and change of position, shape, size, and the like. Specifically, the position correction information is a value such as a coefficient of a function for converting the first image into the second image. Or the information which can calculate the coefficient etc. of such a function may be sufficient. For example, information in which the coordinates in the first image and the coordinates in the second image corresponding to the coordinates are associated with each other may be used as the position correction information. The position correction information may be a coefficient of a coordinate conversion function obtained by substituting these coordinates into a function that performs coordinate conversion into coordinates in the second image.

  As such a function for converting an image, any function may be used. For example, an affine transformation function or the like can be used. In this case, for example, an affine coefficient can be used as the position correction information. A function for performing such coordinate transformation is known as a technique such as so-called morphing. Specific examples of functions will be described later. The alignment described here does not have to aim for perfect matching, but may be any method that makes it easy to determine a matching area between images as described later. The correction parameter storage unit 106 is preferably a non-volatile recording medium, but can also be realized by a volatile recording medium. It may be a non-volatile recording medium or a volatile recording medium.

  The comparison unit 107 corrects at least one of the first image and the image in the region where the first image in the second image is located, using the correction parameter stored in the correction parameter storage unit 106. Then, the first image is compared with the image in the region where the first image in the second image is located. When the first image and the second image are analog signals, the first image and the second image are appropriately converted into digital signals using, for example, an AD converter (not shown). May be. Further, the comparison unit 107 may appropriately convert the first image and the second image into digital signals.

  The comparison unit 107 includes, for example, an image of an area in which the display unit 102 displays the first image in the second image specified as described above (hereinafter referred to as a second area image), a delay buffer, and the like. Corresponding pixels are compared among the pixels constituting the first image read out from 105. A similar comparison is sequentially performed on the pixels in the second region image, usually all the pixels. The comparison between the corresponding pixels of the first image and the second region image described here may not necessarily be a one-to-one comparison between the corresponding pixels. For example, one pixel may be compared with a plurality of pixels, for example, adjacent pixel blocks, or a plurality of pixels may be compared with a plurality of pixels. This is because when the number of pixels of the second region image and the first image received by the receiving unit 101 are different, the pixels do not correspond one-on-one. For example, when comparing one pixel of one image with a plurality of pixels of the other image corresponding to this pixel, for example, a pixel block, the pixel value of one pixel of one image and the pixel value of the plurality of pixels You may compare with representative values, such as an average value and an intermediate value. For example, when comparing a plurality of pixels of one image with a plurality of pixels of the other image corresponding to the plurality of pixels, a representative value such as an average value of pixel values of the plurality of pixels of one image And a representative value such as an average value of pixel values of a plurality of pixels may be compared.

  The comparison between the pixels by the comparison unit 107 is usually performed by comparing pixel information, for example, information such as color intensity of RGB channels and color information values such as luminance information. Specifically, when the second area image and the first image are in a similar relationship and the total number of pixels of both is the same, the relative coordinates of the first image and the second area image are the same. A predetermined pixel value, for example, color information may be compared between the pixels to be processed. Then, such a comparison may be performed between all corresponding pixels. Further, the comparison unit 107 may sequentially perform pattern matching such as a color arrangement pattern of adjacent pixels for corresponding regions of two images instead of comparing color information. Moreover, you may combine these methods.

  Here, in the present embodiment, the comparison unit 107 uses a correction parameter and a function corresponding to the correction parameter to within the region where the first image and the first image in the second image are located. At least one color of the first image is corrected, and the first image is compared with the image in the region where the first image is located in the first image. For example, the color of the first image is corrected, and the corrected first image is compared with the first image taken in the second image, in other words, the second region image. May be. Further, the color of the image in the region where the first image is located in the second image may be corrected, and the first image may be compared with the corrected second image. Further, both the first image and the second image may be corrected, and the corrected first image and the corrected second image may be compared.

  The correction performed here is a correction for color matching for at least one of the first image and the second image using the correction parameter. Specifically, the color correction information in the correction parameter is changed. This is correction of color matching between the used first image and second region image. The color correction information is obtained in advance using the color information of the first image and the color information of the second region image in the second image obtained by capturing the first image, and the correction parameter is obtained. Accumulated in the storage unit 106. In some cases, only one of color correction and position correction may be performed. The color matching correction may be performed on the entire second image, or may be performed only on the second region image. A function or the like used for correction is a function corresponding to the correction parameter stored in the correction parameter storage unit 106.

  Furthermore, in the present embodiment, the comparison unit 107 uses a correction parameter and a function corresponding to the correction parameter to compare the first image and the first image in the region where the first image is located. At least one position of the image is corrected, and the first image is compared with the image in the region where the first image in the first image is located. For example, the comparison between the first image and the second region image by the comparison unit 107 is performed as follows. That is, first, a correlation function between coordinates in the first image photographed in the second image and coordinates in the first image is obtained. Information indicating the correlation function, for example, information such as a coefficient of the correlation function is stored in advance in the correction parameter storage unit 106 as position correction information. In other words, this correlation function is a position correction function used for position correction. Then, using this correlation function or its inverse function, the coordinates in the first image corresponding to a plurality of pixels such as each pixel or each pixel block of the second region image are calculated, and the first image information indicates The information such as the color information of one or a plurality of pixels at the position indicated by the calculated coordinates in the first image is compared with the information such as the color information of the pixels of the corresponding second region image.

  Note that the coordinates in the second image corresponding to each pixel of the first image or a plurality of pixels such as each pixel block are calculated, and the calculated coordinates of the second image indicated by the second image information are Information such as color information of one or a plurality of pixels at the indicated position may be compared with information such as color information of the corresponding first pixel. As described above, using the correlation function, the coordinates in the first image are corrected to the coordinates in the second image, specifically, the coordinates in the second region image, or the second image. Correction for converting the coordinates in the second area image into the coordinates in the first image, here, the first image and the second image, specifically, the second area image Correction for alignment. That is, here, it may be considered that the first image or the second region image is compared while correcting the position. The first image and the second image may be corrected together, and the corrected first image and the corrected second image may be compared.

  The processing for comparing the second region image and the first image as described above may be performed for all the frames of the second image, but at a predetermined or irregular predetermined timing. For example, it may be performed every several frames or every several seconds. In particular, when the first image is an image such as a video conference, it is considered that there is no abrupt movement or the like as described above. Therefore, pixels that match the first image indicated by the first image information once determined This is because it is considered that there is no significant deterioration in image quality even if the region including the image is applied to the process of designating the image that matches the first image from the second image of a plurality of consecutive frames as the transmission region. . As described above, instead of storing information such as correlation function coefficients as position correction information, information capable of calculating information such as correlation function coefficients is stored as position correction information. A correlation function may be obtained from the correction information.

  In addition, here, one or more pixels of the coordinates associated by the correlation function are compared, but the shape, the number of pixels, etc. are preliminarily determined from the first image by correction using the correlation function. An image that is the same as the two-region image may be configured, and the first image configured by the correlation function may be compared with the second region image in units of pixels. Alternatively, an image in which the shape, the number of pixels, and the like are matched with the first image by correction using a correlation function in advance from the second region image, and the second region image configured by this correlation function The first image may be compared on a pixel basis.

  The area where the first image in the second image is located, that is, the position of the second area image in the first image, for example, the display unit 102 in the second image displays the first image in advance. It is preferable to check in which area in the second image the area is displayed, and to set information for designating this area. Information specifying this area may be stored in advance in the correction parameter storage unit 106 in advance, for example, as one of the position specifying information described above. In particular, when the first image and the second region image have a similar relationship, a function that defines the similar relationship can be calculated by using this position designation information. Using the correction, the position of the pixel in at least one of the first image and the second region image is corrected, and the first image can be compared with the second region image.

Note that the region where the first image is located may be detected by image processing or the like. Note that the area where the first image is located in the second image is not only the area where the first image is actually included in the second image, but also the participation in the meeting overlapping the display unit 102. This is an area where the display unit 102 displays the first image when an object such as a person is ignored. For example, when the display area of the display used when the display unit 102 displays the first image is a quadrangle, it is assumed that an object that covers a part of the display, such as a user, is captured in the second image. In addition, the area where the display unit 102 displays the first image is considered to be a rectangle. However, the shape may not completely match depending on the orientation of the display area of the display device with respect to the photographing unit 103.
The comparison unit 107 can usually be realized by an MPU, a memory, or the like. The processing procedure of the comparison unit 107 is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  Using the comparison result of the comparison unit 107, the transmission image configuration unit 108 configures a transmission image in which a region that matches the first image in the second image is set as a transmission region. Specifically, the transparent image construction unit 108 determines whether the comparison unit 107 compares the first image with the image in the region where the first image is located in the second image. A transparent image is configured in which the region in the second image determined to match 108 is set as a transparent region. Specifically, the transmission image constructing unit 108 determines whether or not the compared pixels match each other according to a comparison result between the first image and the second region image of the comparison unit 107. . The comparison result between the pixels here may not be a comparison result between the one-to-one pixels as described above, and may be a comparison result between one pixel and a pixel block, for example. The comparison between the pixels by the transparent image forming unit 108 compares information of predetermined pixels, for example, color information, for the corresponding pixels, and if the difference is within a preset threshold value, the pixels are matched. If it is determined that the pixel is larger than the threshold, it is determined that the pixel is not matched.

  In addition, when the comparison by the comparison unit 107 is a pattern matching comparison, the determination as to whether or not they match may be performed according to the degree of matching, the score, and the like. Then, all the compared pixels and pixel blocks are determined, and an area including the pixels determined to match the first image in the second area image of the second image is determined. Of the second region image, a minute pixel region or the like surrounded by a pixel that matches the first image is considered as noise and is determined to be a pixel that matches the first image. It may be. There are various processes for determining a matching region from the comparison result between images, including the above method, but since it is a known technique, a detailed description thereof will be omitted.

  Then, the transparent image constructing unit 108 constructs a transparent image in which the region determined to match is set as the transparent region. The transmissive area is an area in the second image that is displayed or handled as a transparent area when the second image is displayed, and is an area in which the original second image is not displayed. Alternatively, a region where no image is displayed, or a region where a background image different from the original image or an image having a predetermined pattern is displayed may be considered as a transmissive region. Specifically, the transmissive image construction unit 108 may construct a transmissive image in which a channel image called a so-called alpha channel for designating the transmissive region as an invisible region is added to the second image. The alpha channel image may constitute the same image file as the second image, or may constitute a different image file associated with the second image.

  For example, an alpha channel image or an image file corresponding to a second image which is a JPEG (Joint Photographic Experts Group) file may be attached to the second image file. Further, instead of the alpha channel, a transparent image may be configured by adding information such as a so-called clipping path, which is vector information for designating the outer edge of the region set as the transparent region, to the second image. Note that a technique for masking an image by adding an alpha channel or the like to the image is a known technique, and a description thereof will be omitted. Further, a transparent image from which the color information of the area determined to match may be deleted. In addition, a transparent image in which a matching area is filled with a specific key color may be configured so that the transparent area can be recognized and displayed transparently by a display device or the like that displays the transparent image. However, the addition of the alpha channel is optimal when the original image needs to remain because the original second image is not damaged.

  Further, the transparent image constructing unit 108 may construct a plurality of transparent images obtained by cutting out images of the area separated by the transparent area in the second image. Specifically, in the second image, one or more rectangular areas surrounded by the areas separated by the transparent areas are selected, and the one or more rectangular areas are respectively cut out, that is, clipped, and one or more rectangular areas are selected. Of the transparent image. By clipping the area separated by the transmissive area in this way, it is possible to reduce the transmissive area unnecessary for display and reduce the amount of image data. For each of the transparent images, it is necessary to associate information, for example, coordinates, for specifying the position arranged in the original second image so that the original second image can be reproduced. . The transparent image constructing unit 108 can be usually realized by an MPU, a memory, or the like. The processing procedure of the transparent image forming unit 108 is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  The output unit 109 outputs the transmission image formed by the transmission image configuration unit 108. The output here is a concept including transmission to another apparatus and accumulation in a recording medium or the like. When the image processing apparatus 100 is used in a video conference system or the like that performs a conference in real time, the output unit 109 preferably outputs a transparent image by streaming. There is no limitation on the timing and method for the output unit 109 to output the transmission image. For example, the transmission image may be output every time the transmission image is acquired, or the transmission image may be output when a predetermined amount of transmission image is acquired.

  The output unit 109 converts the transmission image output from the transmission image configuration unit 108 into MPEG, ITU-T H.264, or the like. The compression encoding may be performed using a compression encoding algorithm such as H.264. Further, the transmission image may be converted into an analog signal using a DA (digital analog) conversion unit (not shown) or the like and output. Further, the output unit 109 may add the synchronization information to the transparent image and output it. Further, the output unit 109 may output the captured image and information specifying the transmission area of the captured image in association with each other. Further, when the transparent image is a clipped image of the transparent image as described above, the output unit 109 may output information specifying the position for displaying the clipped transparent image in association with the transparent image. Good.

  Further, the output unit 109 may output a transmission image by adding device identification information or the like of the imaging unit 103 or the like. The output unit 109 may or may not include an output device such as a communication device. The output unit 109 can be realized by output device driver software, or output device driver software and an output device. Note that when the output unit 109 performs compression encoding processing or the like, the output unit 109 may include an MPU (Micro Processing Unit), a memory, or the like for executing these processing. In addition, these processing procedures are usually realized by software, and the software is recorded in a recording medium such as a ROM (Read Only Memory). However, it may be realized by hardware (dedicated circuit).

  Next, the operation of the image processing apparatus will be described using the flowchart of FIG. Here, it is assumed that at least a part of the first image is captured in the second image. Here, as an example, when the first image included in the second image in advance is arranged in a corresponding region of the original first image that is the subject of shooting, the four images of the original first image are displayed. A case will be described in which coordinate information representing the coordinates of one vertex using the coordinate system of the second image is stored in advance in the correction parameter storage unit 106 as position correction information. Further, color correction information which is a coefficient of a function for matching the color of the first image with the color of the second image, particularly the color of the second region image, is stored in the correction parameter storage unit 106. The case will be described.

(Step S201) The receiving unit 101 determines whether or not an input of the first image has been received. If accepted, the process proceeds to step S202. If not accepted, the process returns to step S201.
(Step S202) The display unit 102 acquires the first image received by the receiving unit 101.
(Step S <b> 203) The image processing apparatus 100 stores the first image acquired by the display unit 102 in Step S <b> 202 in the delay buffer 105.
(Step S204) The display unit 102 displays the image information acquired in step S202 on a display (not shown) or the like.

(Step S205) The imaging unit 103 captures a second image including the first image displayed on the display or the like by the display unit 102. The image receiving unit 104 receives information indicating the captured second image and temporarily stores it in a memory or the like (not shown).
(Step S206) The comparison unit 107 acquires the second image received by the image reception unit 104. Also, the first image delayed by the delay buffer 105 is acquired. The first image acquired here is the same first image as the first image included in the second image acquired above, ie, the original first image, delayed by the delay buffer 105. .

(Step S207) The comparison unit 107 compares the first image acquired in step S206 with the second region image in the second image. For example, in the case of a moving image, one frame is compared. Then, the transparent image construction unit 108 determines a matching area according to the comparison result. Details of this processing will be described later.
(Step S208) The transmission image constructing unit 108 constructs a transmission image with an alpha channel in which the region determined to match in step S207 is set as the transmission region. When constructing a transmissive image, the transparency of the boundary between the transmissive area and the non-transparent area or the pixels near the boundary is set to 0% or 100% so that the outer periphery of the non-transparent area is smoothly displayed. Other than that, it is preferable to set it to about 30 to 70%.

  (Step S209) The transmission image construction unit 108 clips the transmission image with the alpha channel formed in step S208 for each image separated in the transmission region. At this time, the transparent region included in each clipped image is made as small as possible. Each clipped image is associated with position information indicating the arrangement in the original captured image, for example, coordinates. Since the technique of clipping for each image separated in the transmissive area is a known technique, description thereof is omitted.

(Step S210) The output unit 109 outputs the transmission image clipped in step S209, and transmits it as an example here. Then, the process returns to step S201.
In the flowchart of FIG. 2, after performing the cutting process in step S <b> 209, a transmission region may be set for each second image that has been cut in step S <b> 209.
In the flowchart of FIG. 2, the process is terminated by powering off or a process termination interrupt.

  Next, the process of step S207 of FIG. 2 will be described using the flowchart of FIG. Here, as an example, it is assumed that the first image and the second region image are color images having R, G, and B channels, and color information of the first image and the second region image is used. A case will be described as an example in which the first image pixel and the pixel in the second region image are compared using the color value, in other words, the color intensity value of each channel. Further, here, the first image and the second image are rectangular images each having p pixels in the x-axis direction (p is an integer of 1 or more) and q pixels in the y-axis direction (q is an integer of 1 or more). The case where it is is demonstrated.

(Step S301) The comparison unit 107 substitutes 1 for a counter m.
(Step S302) The comparison unit 107 substitutes 1 for a counter n.
(Step S303) The comparison unit 107 determines whether or not the pixel located at the coordinates (n, m) of the second image is a pixel of the second region image. For example, the original image in the case where the first image included in the second image stored in the correction parameter storage unit 106 described above is arranged in a corresponding region of the original first image to be imaged. Whether or not the coordinates (n, m) are located in the area connecting the vertices using the coordinate information representing the coordinates of the four vertices of the first image using the coordinate system of the second image Determine whether. When the coordinates (n, m) are located within the area connecting the vertices, it is determined that the pixel is in the area where the first image is captured, that is, the pixel in the second area image. If it is a pixel of the second area image, the process proceeds to step S304, and if it is not a pixel of the second area image, the process proceeds to step S310. The coordinates (n, m) are coordinates in units of pixels in the image. Here, as an example, the upper left coordinates are (0, 0) and the lower right coordinates are (p, q). And

(Step S304) The comparison unit 107 acquires color information of the pixel located at the coordinates (n, m) of the second region image.
(Step S305) The comparison unit 107 corrects the color information of the pixel acquired in step S304 using a color correction function set using the color correction information stored in the correction parameter storage unit 106.

  (Step S306) The comparing unit 107 corrects the coordinates (N, M) (N, M) using the position correction function set using the position correction information stored in the correction parameter storage unit 106 (step S306). However, N is p or less and M is q or less). For example, as described above, when the first image included in the second image is arranged in a corresponding region of the original first image that is the subject of photographing, the four images of the original first image are arranged. The vertex coordinates are expressed using the coordinate system of the second image. When the expressed coordinate information is stored in advance in the correction parameter storage unit 106 as position correction information, these four vertices are added to the position correction function for which a coefficient designated to be used in advance is not determined. By substituting the coordinates, a coefficient of the position correction function is obtained, and a position correction function used for position correction is obtained. Next, coordinates (N, M) corrected by substituting coordinates (n, m) into the position correction function are acquired. The timing at which the comparison unit 107 calculates the coefficient of the position correction function or the like does not matter. In addition, once the coefficient of the position correction function is calculated, the calculation process again may be omitted.

  (Step S307) The comparison unit 107 calculates the color information of the pixel at the coordinates (N, M) of the first image and the color information of the pixel at the coordinates (n, m) of the second region image whose color is corrected in Step S305. And compare. This comparison is performed, for example, by calculating a difference in color intensity values of the same channel between corresponding pixels or an absolute value of the difference. In this comparison, instead of the color information of the pixel at the coordinates (N, M) of the first image, the color information of the pixel block constituted by a plurality of pixels centered on the coordinates (N, M) is used. A representative value may be used.

  (Step S308) The transmission image construction unit 108 uses the comparison result by the comparison unit 107 to determine whether or not the pixel at the coordinates (n, m) of the second region image matches the corresponding pixel in the first image. Determine whether. For example, it is determined whether or not the difference in color intensity for each channel calculated in step S307 is smaller than a preset value, and the difference in color intensity for all channels is determined based on a preset value. Is smaller, it is determined that the pixels match each other. If not, the pixels are determined not to match. In addition, although the case where comparison of pixels and determination of coincidence are performed by obtaining the difference in color intensity of each channel of R, G, and B has been described, this is merely an example, and pixels or pixels and pixel blocks are In addition, comparison between pixel blocks and the like may be performed by any method. If it is determined that they match based on the comparison result, the process proceeds to step S309. If it is determined that they do not match, the process proceeds to step S310.

(Step S309) The transmission image constructing unit 108 temporarily stores, in a memory or the like, a determination result that the (n, m) pixel of the second region image determined in Step S308 is a pixel that matches the first image. The process proceeds to step S310.
(Step S310) The comparison unit 107 increments the value of the counter n by 1.
(Step S311) The comparison unit 107 determines whether or not the value of the counter n is larger than p. If larger than p, the process proceeds to step S312, and if not larger, the process returns to step S303.
(Step S312) The comparison unit 107 increments the value of the counter m by 1.
(Step S313) The comparison unit 107 determines whether the value of the counter m is greater than q. If larger than q, the process proceeds to step S314, and if not larger, the process returns to step S302.

  (Step S314) The comparison unit 107 determines a region that matches the first image in the second region image using the determination result stored in step S309. For example, among the pixels determined to match in step S308, a region that is continuously adjacent is determined, and the coordinates of a plurality of pixels that are the edges of the region are determined in the second region image. Alternatively, it may be stored in a memory or the like as information defining an area that matches the first image. Further, the coordinates of all the pixels determined to match in step S308 may be stored in a memory or the like as information defining an area that matches the first image in the second area image. Then, the process returns to the upper function.

In the flowchart of FIG. 3, the process ends when the power is turned off or the process is terminated.
2 and 3, the R, G, and B channel color information between the pixels is compared, and the pixel of the second region image and the pixel of the first image indicated by the first image information are compared. However, even if it is determined whether or not the pixels match each other using information on other colors between the pixels, such as luminance values, for example. Good. A known technique can be used for such comparison between pixels.

Next, an embodiment of the video conference system using the image processing apparatus according to the above embodiment will be described.
FIG. 4 is a block diagram of the video conference system according to the present embodiment. The video conference system includes a first video conference terminal 10 and a second video conference terminal 20. The first video conference terminal 10 and the second video conference terminal 20 are communicably connected via, for example, a network or a communication line. The first video conference terminal 10 and the second video conference terminal 20 are installed in each conference room so that, for example, a two-way dialogue can be performed between two separate conference rooms. Here, a conference room in which the first video conference terminal 10 is arranged is a first conference room, and a conference room in which the second video conference terminal 20 is arranged is a second conference room.

Here, as an example, the first video conference terminal 10 is assumed to include the image processing apparatus 100 according to the first embodiment.
The second video conference terminal 20 includes a photographing unit 201, a transmission unit 202 that transmits the captured image to the first video conference terminal 10, and a reception unit 203 that receives an image transmitted from the first video conference terminal 10. And a display unit 204 for displaying the received image. In addition, since the structure of the imaging | photography part 201, the transmission part 202, and the display part 204 has the structure similar to the imaging | photography part 103, the output part 109, and the display part 102 of the 1st video conference terminal 10 fundamentally, The description is omitted here.
The receiving unit 203 receives the second image to which the transparent image transmitted from the first video conference terminal 10 is added, that is, the transparent image. The receiving unit 203 can be realized by a device driver for communication means, control software for an apparatus for reading information from a recording medium, or the like.

  FIG. 5 is a conceptual diagram of the video conference system. The first video conference terminal 10 has a display 102a as a display device, and the information processing apparatus 1000 is assumed here except for the display 102a and the photographing unit 103. In addition, the second video conference terminal 20 includes a display 204a as a display device, and the information processing apparatus 2000 is assumed here except for the display 204a and the photographing unit 201. As shown in FIG. 5, in the first video conference terminal 10, the display 102 a that is the display device of the display unit 102 is arranged in the imaging area of the imaging unit 103 so that the display surface faces the imaging unit 103. It is assumed that it is arranged.

  Here, the positional relationship between the first image included in the second image captured by the image capturing unit 103 in advance and the original first image received by the receiving unit 101 is checked and included in the second image. When the first image is placed in the corresponding area of the original first image that is the subject of shooting, the coordinates of the four vertices of the original first image are The coordinate information used is stored in the correction parameter storage unit 106 as position designation information.

  FIG. 6 is a schematic diagram showing the positional relationship between the second image 62 and information on the coordinates of the four vertices of the first image 61 expressed using the coordinate system of the second image 62. For example, here, the coordinates of the four vertices of the first image 61 expressed using the coordinate system of the second image 62 are (x1, y1), (x2, y2), (x3, y3), (x4). , Y4). Here, since the second image 62 shows a case where only a part of the first image 61 displayed on the display 102a is captured, the above four coordinates are all the first coordinates. It is located outside the second image 62.

  However, the second image 62 may be an image obtained by capturing an area including the entire first image displayed on the display 102a. In this case, the coordinates of the four vertices are the coordinates in the second image. Further, the second image 62 may be an image obtained by capturing a part of the first image 61 displayed on the display 102 a and an area other than the first image 61. In this case, some of the coordinates of the above four vertices become the coordinates in the second image 62, and the other coordinates are located outside the second image 62. The unit of coordinates is a pixel.

  Further, color correction information that is a coefficient of a correction function for correcting the difference between the color of the first image 61 and the color of the second region image in the second image 62 is calculated and set in advance. . For example, the first image of a predetermined color is displayed as the first image, the colors at a plurality of positions of the first image, and the second region in the second image obtained by photographing the first image. It is possible to obtain color correction information such as a coefficient of a color correction function by substituting the color of the position considered to correspond to a plurality of positions of the first image into a color correction function used for color correction. it can.

Next, the operation of the video conference system will be described.
First, in the first conference room, the video conference participant A stands between the photographing unit 103 and the display 102a. In the second conference room, the video conference participant B Suppose you were seated in front. Assume that a video conference is started in this state.
First, in the second conference room as shown in FIG. 7, a first image 70 including an image of the upper body of the attendee B photographed by the photographing unit 201 of the second video conference terminal 20 is sent from the transmitting unit 202. It is transmitted to the first video conference terminal 10 via the network.

  In the first video conference terminal 10 of the first conference room, the reception unit 101 receives the first image. Here, it receives from the second video conference terminal 20. The display unit 102 acquires the first image as shown in FIG. 7 received by the receiving unit 101 and displays the first image on the display 102a. Next, the imaging unit 103 captures a second image including the image of the attendee A in the first conference room. The image receiving unit 104 receives the captured second image. The received second image may be temporarily stored in a memory (not shown) or the like.

  FIG. 8 is a diagram illustrating an example of the second image 80. In the first meeting room, the shooting area of the shooting unit 103 is located within the display area of the display 102a. The display 102a displaying the first image including the B image is taken as a background.

  Here, when the second image captured by the imaging unit 103 is transmitted as it is from the output unit 109 to the second video conference terminal 20 and displayed on the display 204a of the display unit 204, the participant in the second conference room B will see on the display a second image in which the first image of the subject being photographed is displayed in the background. That is, the first image taken in the second conference room is fed back and displayed on the display 204a in the second conference room. For this reason, it cannot be said that an appropriate image is displayed for a video conference, and there is a risk of hindering smooth progress of the conference. For example, in the second meeting room, the participant B may be concerned about his / her own image fed back and displayed on the display 204a, distracting attention, and the meeting may not be performed efficiently. .

For this reason, in this Embodiment, the process which transmits the image which made such an image fed back invisible to another meeting room is performed.
First, the receiving unit 101 receives a first image and outputs it to the display unit 102, and at the same time, the same first image is accumulated in the delay buffer 105. The comparison unit 107 acquires the second image received by the image reception unit 104 and is the same as the first image captured in the second image that is delayed for a predetermined time by the delay buffer 105. One image is acquired from the delay buffer 105.

  Next, for each pixel of the second image, it is determined whether or not it is an image in an area where the first image in the second image is located, that is, whether or not it is a pixel of the second area image. To do. That is, whether or not the pixel is in a region defined by the coordinates (x1, y1), (x2, y2), (x3, y3), (x4, y4) of the four vertices as shown in FIG. Determine whether. Here, all the pixels of the second image are determined to be pixels of the second region image.

  And about the pixel judged to be a pixel of this area | region, the color information of the said pixel, the value of the color intensity of R, G, and B here is acquired. Next, the color intensity value is corrected by a color correction function using the color correction information stored in the correction parameter storage unit 106.

  As a color correction function used here, for example,

R = a ₁ r + a ₂ g + a ₃ b,
G = b ₁ r + a ₂ g + a ₃ b,
B = c ₁ r + a ₂ g + a ₃ b

The color correction function is used.

Here, the pixel value of the second image, that is, each color intensity value is (r, g, b), and the corrected pixel value, that is, each color intensity value is (R, G, B). The a _{1 to} a ₃ , b _{1 to} b ₃ , and c _{1 to} c ₃ are color correction function coefficients, that is, color correction information stored in the correction parameter storage unit 106. Therefore, the corrected color information (R, G, B) is obtained by substituting the color information of the pixel determined to be the pixel of the second region image into r, g, b of this color correction function. Can do.

  Next, the comparison unit 107 uses the coordinates of the four vertices of the second area image in the second image, which is the position correction information stored in the correction parameter storage unit 106, for the position correction specified in advance. The coefficient to be substituted is substituted into a position correction function for which the coefficient is not specified. Then, the position correction function is obtained by calculating the coefficient of the position correction function for calculating the coordinates of the first image corresponding to the second region image.

The position correction function used here is, for example,

X = ax + by,
Y = cx + dy

  Is a function represented by Here, the position of the pixel constituting the second region image in the second image is (x, y), the position of the pixel in the original first image is (X, Y), and a, b, c , D are coefficients of the position correction function.

  In this case, by substituting the coordinates (x1, y1), (x2, y2), (x3, y3), (x4, y4) of the above four vertices into x and y of the above function, a, b , C, d can be calculated. Thereby, the position correction function used when making the image fed back invisible can be calculated | required. Note that the values of a, b, c, and d may be given as position correction information from the beginning. Further, the correction function itself that gives the values of a, b, c, and d may be used as the position correction information.

  Then, using this position correction function, the coordinates specifying the pixel or pixel block in the first image corresponding to the pixel determined to be the pixel of the second area information are acquired. Specifically, by substituting the coordinates of the pixel determined to be the pixel of the second area information into x and y of the acquired position correction function, the corresponding coordinate (X, Y) in the first image is obtained. Obtainable.

  Next, the color information of the pixel or pixel block located at this coordinate in the first image is acquired and compared with the color information obtained by color-correcting the color information of the pixel of the corresponding second region image. Specifically, the color information of the pixels, specifically, the color intensity values of the R, G, and B channels are compared, and if all the differences are smaller than a predetermined value, it is determined that they match.

  For example, the value of each channel (R, G, B) of the pixel determined to be a pixel in the second area image is (230, 13, 56), and the coordinates of the pixel in the second area image are It is assumed that the value of each channel (R, G, B) of the pixel located at the coordinates in the first image obtained by correction by the position correction function is (231, 10, 57), If the threshold values of all the channels are all “5”, the absolute value of the difference between all the channel values is smaller than 6 in this case. It is determined that the pixels match each other. For this reason, the pixel determined to be a pixel in the second region image is determined to be a pixel in the first image. Then, the pixels determined to be the pixels in the second region image are temporarily stored in a memory or the like (not shown) as the pixels matching the pixels in the first image. The threshold value may be set in advance or may be set as appropriate.

  Conversely, for example, the value of each channel of (R, G, B) of the pixel determined to be a pixel in the second region image is (230, 13, 56), and the pixel in the second region image If the value of each channel of (R, G, B) of the pixel located at the coordinates in the first image obtained by correcting the coordinates of by the position correction function is (238, 10, 50). In this case, since the absolute value of the difference between the values of the R and B channels is “6” or more, it is determined that the pixels do not match, and the pixels determined to be pixels in the second region image are It is determined that it is not a pixel in the first image. That is, this pixel is considered to be a pixel or the like that captured the participant A located in front of the display 102a. Such a process is performed for all the pixels of the second image.

  Thereafter, the transparent image constructing unit 108 reads out a pixel that matches the pixel in the first image stored in the memory or the like, and matches the area including this pixel with the first image in the second image. Decide on an area. For example, a region where pixels are continuous is sequentially detected. Then, the coordinates and the like of a plurality of pixels serving as the edges of this region are acquired as information defining a region in the second image that matches the first image, and are temporarily stored in a memory or the like (not shown).

  FIG. 9 is a diagram showing a region 90 that matches the first image in the second image 80. Next, the transmission image construction unit 108 reads information defining an area that matches the first image in the second image, configures an alpha channel in which this area is set as the transmission area, and configures the second image. Append to FIG. 10 is a diagram schematically showing a state in which the alpha channel 95 is added to the second image 80. An image to which the alpha channel is added is a transmission image. It is assumed that the alpha channel 95 constitutes the same file as the second image 80. However, as long as it is associated with the second image 80, the alpha channel images do not have to form the same file.

  Next, the transparent image constructing unit 108 clips, that is, cuts out an area separated by the transparent area. The alpha channel is also retained in this clipped second image. Further, information indicating where each of the clipped second images is arranged in the second image before clipping is obtained, and is associated with the clipped second image in a memory or the like (not shown). Keep it. Here, clipping is performed after the alpha channel is configured, but the alpha channel may be configured after clipping. Since the process of performing clipping based on information such as the transparent area is known, the description thereof is omitted.

  FIG. 11 is a diagram showing second images 91 and 92 obtained by clipping. FIG. 12 is a clipping image management table for managing the second images 91 and 92 obtained by clipping and information indicating positions where the respective images are arranged, in this case, coordinates. In the clipping image management table, “coordinate” indicates the upper left coordinate of the clipping image in the original second image. “Image” indicates an image obtained by clipping.

The output unit 109 associates the second images 91 and 92 obtained by clipping with coordinate information indicating the positions where these images are arranged, and transmits the information to the second video conference terminal 20.
By transmitting the clipped image in this way, it is possible to delete the transparent region in the second image, that is, the image in the region that does not need to be displayed, and reduce the amount of information of the second image to be transmitted. . As a result, it is possible to efficiently transmit and receive images within a limited communication band, and it is possible to reduce the occurrence of image delays, disturbances, frame dropping, and the like.

The second video conference terminal 20 receives the second images 91 and 92 obtained by clipping and coordinate information associated therewith, and temporarily stores them in a memory or the like (not shown). Then, the display unit 204 arranges the second images 91 and 92 at the positions indicated by the coordinate information associated with the second images 91 and 92, and displays, for example, a preset background image or the like in the transparent area indicated by the alpha channel. The arranged display image is constructed and displayed on the display 204a. FIG. 13 is a diagram illustrating a display example on the display 204 a of the second video conference terminal 20. Thereby, a video conference can be realized in both directions.
In this specific example, the output unit 109 outputs the clipped second image. However, when the communication band is wide, the output unit 109 outputs the second image without clipping. It may be.

  As described above, according to the present embodiment, an image that matches the first image displayed on the display unit 102 from the second image captured by the imaging unit 103 is set as the transmission region. It is possible to prevent the image displayed by the unit 102 from being included in the second image captured by the imaging unit 103 and being output. Thereby, for example, an image obtained by removing an image of a participant in a conference in another conference room displayed on the display unit 102 in the background of the participant from an image obtained by the imaging unit 103 capturing a participant such as a video conference. Can be sent to video conference terminals in other conference rooms. As a result, images taken in each conference room can be prevented from being fed back and displayed on the display in the same conference room, etc., and confusion caused by the fed back images can be suppressed, making the conference more efficient. Can be performed.

  Further, correction for color matching and positioning between the first image and the first image included in the second image is performed using correction parameters such as position correction information and color correction information. By comparing the first image and the first image included in the second image, it is possible to compare the images with high accuracy. For example, in general, the first image and the image that is once displayed on the display 102a or the like and the image captured by the image capturing unit 103 have colors depending on the characteristics of the device used, the influence of illumination, and the like. In this embodiment, images can be compared by correcting such a color difference.

  In addition, due to the arrangement of the photographing unit 103 and the display 102a, the lens characteristics of the photographing unit 103, and the like, the first image in the second image is greatly deformed in size and shape compared to the original first image. However, it is difficult to compare the two pixels in correspondence with each other. However, in the present embodiment, by performing position correction, the position in the first image is included in the second image. Since the position in the first image can be accurately aligned and matched and compared, accurate comparison is possible.

In the above specific example, the second video conference terminal 20 may have the same configuration as that of the first video conference terminal 10.
In the specific example, the display unit 102 and the shooting unit 103 of the first video conference terminal 10 are each one, and the shooting unit 201 and the display unit 204 of the second video conference terminal 20 are also one. However, there may be a plurality of these. For example, as shown in FIG. 14, the first video conference terminal 10 is provided with six display units (not shown) similar to the display unit 102, and the six displays 102 a that are the respective display devices are arranged in an octagonal shape. It is arranged side by side on the six sides of the area, and each of the six photographing units 103 is arranged at the top of each display 102a, and a conference table or the like is arranged at the center of the octagonal area. Good.

  In this case, the second video conference terminal has the same configuration, and the second image captured by each imaging unit 103 of the first video conference terminal 10 is a display device of six display units of the second video conference terminal (not shown). The first images captured by the six imaging units of the second video conference terminal (not shown) are displayed on the six displays 102a of the first video conference terminal 10, respectively. It is preferable to do so. By providing a plurality of displays for displaying the images of the participants in this way, it is possible to realize a conference with a large number of people and a sense of reality.

  In the above specific example, the pixels in the second region image and the pixels in the first image corresponding to the coordinates obtained by coordinate conversion of the coordinates of the pixels with the position correction function are appropriately compared. However, by correcting the position of the entire second region image using the position correction function in advance, an image having the same shape as the first image is formed, and each pixel of the first image You may make it compare with a pixel block etc. In addition, the position of the entire first image is corrected in advance using a position correction function to form an image having the same shape as the second area image, and the configured first image is converted into each pixel of the second area image. Or a pixel block or the like.

(Embodiment 2)
The image processing apparatus according to the present embodiment includes the above-described configuration for acquiring the correction parameter in the image processing apparatus according to the first embodiment.
FIG. 15 is a block diagram illustrating a specific example of the image processing apparatus according to the present embodiment.
The image processing apparatus 300 includes an image storage unit 301, a reception unit 101, a display unit 302, a photographing unit 303, an image reception unit 104, a delay buffer 105, a correction parameter storage unit 106, a comparison unit 107, a transmission image configuration unit 108, and an output unit. 109, a captured image storage unit 304, a landmark image detection unit 305, a correspondence information acquisition unit 306, a correspondence management information storage unit 307, and a correction parameter configuration unit 308.

  The correction parameter configuration unit 308 includes color information acquisition means 3081, color correction information acquisition means 3082, and position correction information acquisition means 3083. Since the reception unit 101, the image reception unit 104, the delay buffer 105, the correction parameter storage unit 106, the comparison unit 107, the transmission image configuration unit 108, and the output unit 109 have the same configurations as those in the first embodiment, description will be made. Omitted.

  The image storage unit 301 stores a number of pattern images, which are images in which one or more, preferably a plurality of landmark images are arranged at different positions. Between the plurality of pattern images, the positions at which the plurality of mark images are arranged are the same. Further, in the plurality of pattern images, the appearance patterns of the colors of the mark images arranged at different positions are different from each other. A landmark image is an image that serves as a landmark. The mark described here is a mark for identifying a position in the image. The mark image is an image constituting a part of a pattern image to be described later. Specifically, the mark image is used when obtaining a correspondence between a position in an image displayed by the display unit 302 (to be described later) and a position in an image displayed by the display unit 302 in an image shot by the shooting unit 303. It is an image used for. The landmark image is an image having a color and a shape that can be detected separately from the background or the like. When the position where the mark image is displayed is the center position of the mark image, the shape of the mark image is preferably circular because the center position does not easily change even if the mark image is deformed. However, other shapes such as a polygon may be used. The size of the mark image is not limited.

  Each mark image usually has any one of a plurality of colors. The color of the landmark image is an image that becomes the background of the landmark image in an image obtained by photographing the image including at least a part of the pattern image displayed on the display unit 302 by the photographing unit 303 (hereinafter referred to as a photographed image). A color that can be identified for the background image. Each mark image is preferably a uniform color, but may have some variation as long as it can be identified. The mark image and the pattern image may be a vector image or a raster image. The area where the mark image of the pattern image is not arranged is usually a background image.

  The pattern image is, for example, an image configured by arranging a plurality of landmark images at predetermined positions of a background image having the number of pixels corresponding to the number of pixels of the image that the display unit 302 normally displays. There is no limitation on the arrangement interval or positional relationship between the mark images in the pattern image. In each pattern image, the position where each mark image is arranged is normally designated in advance. The position where the mark image is arranged is specified by information indicating the position in the image that the display unit 302 normally displays, for example, coordinates. The position where the mark image is arranged may be specified by information indicating the position in the display area of the display unit 302, such as coordinates. The mark image may be stored in the image storage unit 301 in association with information specifying a position to be displayed or a position to be arranged.

  The pattern image may be considered as a plurality of landmark images associated with information indicating the display position in the image displayed by the display unit 302. In this case, when the display unit 302 performs display, a pattern image for display in which the plurality of mark images are arranged at respective display positions may be configured to display the pattern image. Here, as described above, between the pattern images, the positions at which the plurality of mark images are arranged coincide with each other, and the color of the mark image displayed at the same position appears, that is, the appearance pattern. However, the mark image is arranged in each pattern image so as to be different for each position.

  In other words, the color appearance pattern is a pattern of color change of the mark image. Specifically, the appearance pattern is a permutation or combination of colors of a plurality of mark images at positions where the mark images are arranged among a plurality of pattern images. For example, the appearance pattern is a permutation in which the color of the mark image appears at the arrangement position of each mark image when a plurality of pattern images are arranged in a predetermined order. In addition, for example, the appearance pattern may be a combination of colors of the extracted landmark images when a plurality of landmark images are extracted from the plurality of pattern images at the position where each landmark image of the pattern image is arranged. Good. For example, in the present embodiment, when displaying a plurality of pattern images in a preset order, a plurality of landmark images whose appearance patterns of colors of the landmark images displayed at different positions are different from each other, Each pattern image is made to have.

  In this way, although the details will be described later, the landmark image is arranged using the appearance pattern of the color of the landmark image for each position where the landmark image is arranged in the image photographed by the photographing unit 303. Can be identified. Note that replacing the color information with a numerical value and acquiring the appearance pattern of the numerical value is considered to acquire the color appearance pattern here. Here, as an example, it is assumed that the color of each mark image in each pattern image is one of two colors. Further, the combination of the two colors of the mark images between the pattern images may be the same, or may be different. In particular, by making them all different, it is possible to determine which pattern image is an image obtained by photographing which pattern image by determining the combination of the colors.

  As will be described later, in the case where the mark image in each pattern image has one of two colors, in order to be able to identify the position where each mark image is arranged, When the number of landmark images arranged in the image is A (A is an integer of 2 or more) and the number of pattern images is B (B is an integer of 2 or more),

2 ^B ≧ A

  A plurality of pattern images satisfying the above relationship are stored in the image storage unit 301. The image storage unit 301 is preferably a non-volatile recording medium, but can also be realized by a volatile recording medium.

  The display unit 302 reads the plurality of pattern images stored in the image storage unit 301 in the display unit 102 that displays the first image described in the above embodiment, and switches and displays them. Is. The plurality of pattern images are preferably displayed in a predetermined order. However, each pattern image is selected from the pattern image displayed on the display unit 302 or the pattern image included in the image captured by the image capturing unit 303. In the case where identification is possible, the order in which the display unit 302 displays each pattern image does not matter. Here, the orientation of the display surface of the display device of the display unit 302 is not changed between the display of the pattern image and the display of the first image.

  The image capturing unit 303 captures an image including the pattern image displayed on the display or the like by the display unit 302 in the image capturing unit 103 that captures the second image described in the first embodiment. Is. The imaging unit 303 may capture an image of an area including other areas as well as an area where the display unit 302 displays an image. Further, when the display unit 302 switches and displays a plurality of pattern images, each image including the plurality of pattern images that are switched and displayed on the display unit 302 is captured. Further, the photographing unit 303 may photograph only a part of the pattern image displayed on the display unit 302 including a plurality of landmark images. The photographing unit 303 may photograph a moving image including a plurality of frame images, or may photograph a still image.

  As described above, when a plurality of pattern images are switched and displayed, the imaging unit 303 may capture a still image every time the pattern image is switched, or a moving image including all the switched pattern images. You may shoot. For example, the imaging unit 303 may receive information indicating that the displayed pattern image is switched from the display unit 302, and may capture a still image when the switching information is received. Further, the photographing unit 303 may add information indicating that the switching has been performed to a frame, a header, or the like of the moving image being photographed. By referring to this information, it is possible to determine the timing when the pattern image is switched. The photographing unit 303 stores the photographed image in the photographed image storage unit 304. The imaging unit 303 may configure information indicating the display order of the captured pattern images and store the information in the captured image storage unit 304 or the like in association with the captured image that is the captured image. Here, the photographing unit 303 is used by fixing the photographing region or the like when the pattern image is photographed and when the second image is photographed.

  The captured image storage unit 304 stores an image including a pattern image captured by the capturing unit 303 and displayed on the display or the like by the display unit 302. Specifically, the captured image storage unit 304 stores an image including a pattern image received by the image receiving unit 104. The captured image storage unit 304 is preferably a non-volatile recording medium, but can also be realized by a volatile recording medium.

  The landmark image detection unit 305 detects a landmark image in the image photographed by the photographing unit 303. When the image capturing unit 303 captures a pattern image displayed by switching to the display unit 302, the landmark image detection unit 305 captures each still image including each pattern image displayed by the display unit 302 captured by the image capturing unit 303. Each mark image in the image is detected. Alternatively, the mark image detection unit 305 extracts one frame from the portion where each pattern image is photographed from the moving images obtained by photographing the pattern images displayed on the display unit 302 by the photographing unit 303, respectively. Each landmark image in the image is detected. Instead of extracting one frame, a plurality of frames may be extracted, and a composite image for one frame obtained by averaging the plurality of frames may be configured in advance.

  As a result, the mark image detection unit 305 only needs to be able to detect the mark image from all the pattern images. For example, instead of detecting the mark image for all the pattern images, the mark image detecting unit 305 detects the mark image for one pattern image, and the like. As for the pattern image, the mark image may be detected at the same position as the detected position. Even in such a case, it is considered that the mark image is detected for each pattern image.

  The landmark image detection unit 305 reads, for example, information on a plurality of colors used in the landmark image accumulated in advance in a memory or the like, and for each pixel in a still image including each pattern image or one frame image. An area in which pixels that are the same as or close to the color of the mark image are judged to be the same as any of the colors used in the mark image, and pixels that are the same as or close to the color of the mark image Is detected, the image in this area is detected as a landmark image in the image captured by the image capturing unit 303.

  The mark image detection unit 305 accumulates information for specifying the detected mark image in, for example, a storage medium such as a memory (not shown). The information for designating the detected landmark image may be information that can designate the landmark image. For example, the coordinate information of each pixel constituting the landmark image, the coordinate information of the pixel representing the landmark image, for example, the center pixel, etc. Coordinate information for designating the outline of the mark image. The landmark image detection unit 305 may store identification information for management in association with each landmark image in a memory or the like.

  When the combination of the two colors used as the mark images of the plurality of pattern images is different for each pattern image, the mark image detection unit 305 is used for the mark image detected from each pattern image. It is also possible to acquire a combination of colors, identify a pattern image in the captured image using the combination of colors, or acquire an order in which the pattern images in the captured image are arranged. For example, determination information in which each pattern image is associated with information indicating a combination of two colors used for each pattern image is prepared in advance, and a captured image is selected from the color combinations of the information. Search for a color combination that matches the color combination used in the landmark image, and determine or identify that the pattern image associated with the matching combination is a pattern image in the captured image You can do it.

  The mark image detection unit 305 can usually be realized by an MPU, a memory, or the like. The processing procedure of the mark image detection unit 305 is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  In the correspondence management information storage unit 307, the correspondence between the position where the plurality of landmark images are arranged in the plurality of pattern images stored in the image storage unit 301 and the appearance pattern of the color at the arranged position is displayed. Correspondence management information which is information to be stored is stored. For example, the position where the mark image is arranged may be expressed by the coordinates of the position where the mark image is arranged in each pattern image. Alternatively, the mark image may be represented by coordinates in the display area of the display unit 302. The information indicating the position where the mark image is arranged may be information that can specify the position of the mark image, such as coordinate information of the center of the mark image. Moreover, the information etc. which designate the outline etc. of a mark image may be sufficient.

  The appearance pattern of the color at the position where the mark image is arranged is a pattern in which the color of the mark image appears, in other words, a change pattern. For example, the appearance pattern is a permutation or combination of colors at positions where the mark images are arranged. For example, the appearance pattern is a permutation in which the color of the mark image displayed at the display position of each mark image appears when a plurality of pattern images are arranged in a predetermined order. Further, for example, the appearance pattern may be a combination of colors of the acquired landmark images when the landmark images are acquired from the positions where the respective landmark images are arranged in a plurality of pattern images.

  Note that the color appearance pattern may be information in which color values, names, and the like are changed and arranged in order. For example, the appearance pattern is represented by red, yellow, recording, red, blue, or the like. Alternatively, the information may be information representing the order of change of values by replacing color information or the like with numerical values, and two colors used as a mark image in one pattern may be expressed by binary values. It may be a pattern in which binary values appear.

  For example, in the first pattern image, the red mark information is “0” and the blue mark information is “1”. In the second pattern image, the green mark information is “0” and the blue mark information is In the third pattern image, when the yellow mark information is “0” and the blue mark information is “1”, the appearance patterns are “011”, “101”, “0” is represented by a combination of three and “1” by five. The corresponding display position and appearance pattern are managed as two attribute values of one record in the database, for example. The correspondence management information storage unit 307 is preferably a nonvolatile recording medium, but can also be realized by a volatile recording medium.

  The correspondence information acquisition unit 306 acquires correspondence information that is information indicating the correspondence between the position of the landmark image in the image photographed by the photographing unit 303 and the display position designated in advance of the landmark image. For example, the correspondence information acquisition unit 306 acquires a color appearance pattern for a landmark image detected at the same position from each landmark image detected by the landmark image detection unit 305, and the appearance pattern of the color is acquired. Correspondence information indicating the correspondence between the position in the image photographed by the photographing unit 303 and the display position of the mark image in a plurality of pattern images having a color appearance pattern that matches the color appearance pattern is acquired. The match here may be a complete match or a partial match.

  The correspondence information acquisition unit 306 acquires, for example, color information of each mark image detected in the image captured by the mark image detection unit 305 for each pattern image. Then, when the images obtained by capturing the pattern images displayed on the display unit 302 are arranged in a predetermined order specified in advance, detection is performed at substantially the same position among the mark images detected by the mark image detection unit 305. The appearance pattern of the color for the marked landmark image is acquired. As described above, the color appearance pattern may be information in which the order of change of the color value, name, etc. is arranged, or the order of change of the value by replacing the color information, etc. with a numerical value. It may be a pattern in which two colors appear when two colors used as a mark image in one image are represented by binary values.

  Then, the correspondence information acquisition unit 306 acquires information on the position where the landmark image is displayed in one or more pattern images having a color appearance pattern that matches the color appearance pattern. Specifically, a pattern image that matches the appearance pattern of the color of the landmark image in the image captured by the imaging unit 303 acquired by the correspondence information acquisition unit 306 from the correspondence management information stored in the correspondence management information storage unit 307. The position information of the landmark image associated with the appearance pattern of the color of the landmark image is acquired. Then, correspondence information indicating correspondence between the position information where the acquired landmark image is displayed and the positional information of the landmark image detected by the landmark image detection unit 305 having the same color appearance pattern is acquired.

  Specifically, the correspondence information is information having a pair of position information where the corresponding mark image is displayed and position information of the mark image in the captured image detected by the mark image detection unit 305. The position information here is, for example, coordinate information. Note that it is not always necessary to acquire correspondence information for the positions of all landmark images detected by the landmark image detection unit 305. Also, the correspondence information acquisition unit 306 can be usually realized by an MPU, a memory, or the like. The processing procedure of the correspondence information acquisition unit 306 is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  The correction parameter configuration unit 308 configures correction parameters using the correspondence information acquired by the correspondence information acquisition unit 306 and accumulates the correction parameters in the correction parameter storage unit 106. The correction parameters described here are correction parameters similar to the correction parameters described in the first embodiment, and are, for example, position correction information and color correction information. The position correction information and the color correction information may be a position correction function coefficient, a color correction function coefficient, or the like, or may be a position correction function or a color correction function itself. The correction parameter configuration unit 308 can usually be realized by an MPU, a memory, or the like. The processing procedure of the correction parameter configuration unit 308 is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  The color information acquisition unit 3081 includes color information that is information related to colors of a plurality of landmark images in an image captured by the imaging unit 303 and an image captured by the imaging unit 303 indicated by the correspondence information acquired by the correspondence information acquisition unit 306. And the color information of the landmark image in the pattern image corresponding to the plurality of landmark images. The color information acquisition unit 3081 may acquire color information regarding a position where one landmark image is displayed, or may acquire color information regarding a position where a plurality of landmark images are displayed. However, the color information acquisition unit 3081 acquires color information from one pattern image and an image obtained by the imaging unit 303 capturing the pattern image displayed on the display unit 302.

  The color information may be a part of each mark image, for example, color information of one pixel or the like, or color information calculated by using a least square method or the like from the color information of all the pixels in the mark image. There may be. The color information acquisition unit 3081 can be usually realized by an MPU, a memory, or the like. The processing procedure of the color information acquisition unit 3081 is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  The color correction information acquisition unit 3082 acquires color correction information used for correcting the color of the image received by the image reception unit 104 using the color information acquired by the color information acquisition unit 3081. Specifically, the color correction information is a value such as a coefficient of a function used for color correction. It may also be a function itself. For example, the color correction information can be calculated by substituting the value of the color information acquired by the color information acquisition unit 3081 into a function for converting the color. As these functions, the same color correction function as in the first embodiment is used.

  The color correction described here is finally correction for matching the colors of the first image and the color of the second region image in the second image. However, here, as the color correction information used for this correction, the color correction used for correcting the color of the image photographed by the photographing unit 303 with the original of the pattern image photographed by the photographing unit 303 is used. Information shall be used. The color correction in this case may be correction of relative color between the photographed image and the original pattern image. Even if the photographed image is color-matched with the original pattern image, the original pattern image is corrected. May be a correction for color matching the captured image.

  The color correction information acquisition unit 3082 accumulates the acquired color correction information in the correction parameter storage unit 106. The color correction information acquisition unit 3082 can be usually realized by an MPU, a memory, or the like. The processing procedure of the color correction information acquisition unit 3082 is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

The position correction information acquisition unit 3083 acquires position correction information used for correcting the position of the image received by the image reception unit 104 using the position information acquired by the correspondence information acquisition unit 306.
The position correction described here is finally correction for aligning the position of the first image and the position of the second region image in the second image. However, here, as the position correction information, it is assumed that the position correction information used for correction for matching the image captured by the image capturing unit 303 with the original pattern image captured by the image capturing unit 303 is used. However, the position correction in this case may be a relative correction between the captured image and the original pattern image, and even if the captured image is corrected to match the original pattern image, the original pattern image is Correction may be made to match the captured image.

  This position correction is usually coordinate conversion between the coordinates of the captured image and the coordinates of the original pattern image. Specifically, the position correction information is a coordinate conversion of each pixel constituting the image displayed on the display unit 302 into a position in the image displayed on the display unit 302 in the image captured by the shooting unit 303. Function coefficients for converting the coordinates of the pixels in the image displayed by the display unit 302 to the positions in the image displayed by the display unit 302 in the image captured by the imaging unit 303, etc. Is the value of However, the position conversion function itself may be used. For example, by substituting the coordinate information of the position of the landmark image in the pattern image indicated by the correspondence information and the coordinate information of the position of the landmark image in the corresponding captured image into the function for coordinate transformation of the image, Correction information can be calculated.

  As a function for coordinate transformation, for example, there is an affine transformation function. In this case, the position correction information is an affine coefficient. Any function is acceptable. A function for performing such coordinate transformation is known as a technique such as so-called morphing. The position correction information acquisition unit 3083 accumulates the acquired position correction information in the correction parameter storage unit 106. The position correction information acquisition unit 3083 can usually be realized by an MPU, a memory, or the like. The processing procedure of the position correction information acquisition unit 3083 is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

Next, the operation of the image processing apparatus 300 will be described. Of the operations of the image processing apparatus 300, the first image is displayed, and the region that matches the first image is deleted from the second image obtained by capturing the region including the first image. Since is the same as that of the first embodiment, detailed description thereof is omitted.
Hereinafter, the operation of acquiring the correction parameter of the image processing apparatus 300 will be described with reference to the flowchart of FIG. Here, p (p ≧ 2, p is an integer) number of pattern images having a plurality of mark images having two colors at different positions are prepared, and each mark image is between each pattern image. Are displayed at the same position, and when each pattern image is switched or arranged or displayed in a predetermined order, the appearance patterns of the colors of the mark images displayed at the same display position are mutually different for each position. Assume that they are set differently. In each pattern image, the position where the landmark image is displayed, the color used as the landmark image, and the background color are specified in advance. Normally, black or the like is used as the background color.

(Step S1601) The display unit 302 substitutes 1 for a counter m.
(Step S1602) The display unit 302 reads the mth pattern image from the image storage unit 301 and displays it on a display or the like.
(Step S1603) The photographing unit 303 photographs the mth pattern image displayed on the display unit 302. The photographing unit 303 may photograph the pattern image as a still image or a moving image. The image captured by the imaging unit 303 is received by the image receiving unit 104 and accumulated in the captured image storage unit 304. Note that the photographing unit 303 may accumulate information indicating the order of photographing or information for identifying the photographed pattern image in association with the photographed image.

(Step S1604) The display unit 302 increments the counter m by 1.
(Step S1605) The display unit 302 determines whether m is larger than p. If m is larger than p, the process proceeds to step S1606, and if not larger, the process returns to step S1602.
(Step S1606) The image processing apparatus 300 detects a landmark image from the images captured by the capturing unit 303, that is, from the captured images stored in the captured image storage unit 304. Details of the process of detecting the landmark image will be described later.
(Step S1607) The image processing apparatus 300 acquires a correction parameter. The process for acquiring the correction parameter will be described later. Then, the process ends.

In the flowchart of FIG. 16, the trigger, timing, and the like at which the image processing apparatus 300 starts the above processing are not questioned. For example, it may be performed immediately before the first image is displayed by the display unit 302, or when the display of the display unit 302, the photographing unit 303, or the like is installed.
In the flowchart of FIG. 16, the process is terminated by power-off or a process termination interrupt.

Next, details of the process of detecting the landmark image shown in step S1606 will be described using the flowchart of FIG.
(Step S1701) The mark image detection unit 305 substitutes 1 for the counter n.
(Step S1702) The mark image detection unit 305 reads out and acquires the m-th captured image from the captured image storage unit 304.

  (Step S1703) The mark image detection unit 305 performs a process of detecting the positions of a plurality of mark images from the m-th captured image. For example, the color of each pixel of the landmark image is sequentially compared with a plurality of colors that are designated in advance as the landmark image. And an area of a pixel adjacent to a color determined to be the same color as that pixel is determined as an area of one mark image. It is determined that the non-adjacent areas are areas of different mark images. Judgment of the color of the pixel for detecting the landmark image is made by determining whether or not the color component of the pixel exceeds these thresholds by providing a threshold for the information of the color component such as the RGB value, for example. It is preferable to do.

  The color component itself is missing between the color of the pixels that make up the landmark image for the captured image that is displayed on the display, etc., and the other image is taken, and the color of the pixels that make up the landmark image in the original pattern image It is possible that the value of the color component will change greatly even if it is not reversed or reversed, so by setting the color range using the threshold value etc. in this way, It is preferable to determine whether or not the pixel is regarded as the same color as the original color of the landmark image. Further, in such processing, the color component of the pixel is binarized to “0”, “1”, etc. by the threshold value, and it is determined from the binarized information whether the pixel is in the mark image. Good. The process for detecting the mark image is not limited to the above process. For example, by using a technique such as pattern matching for detecting an image having a specific color gamut or a similar shape, a position that is considered to be the same as the mark image as described above is detected. Also good. Since such a technique is a known technique, a description thereof will be omitted.

  (Step S1704) The mark image detection unit 305 acquires the position information of each mark image detected in step S1703 and the color information of the mark image and outputs them in association with each other, such as a memory (not shown). Stored in the storage medium. At this time, it is preferable to output the identification information for identifying each mark image in association with each other. As the information on the position of the mark image, the center coordinates in the area of the mark image in the photographed image are used. However, the information on the position of the mark image may be information that defines the outline of the mark image, the coordinates of all or some of the pixels of the mark image, etc., as long as the information can specify the position of the mark image. . The color information of the mark image may be information on the color of a specific pixel in the mark image, for example, the center pixel in the area of the mark image, or all or some of the pixels in the mark image. It may be information such as a representative value of each color, such as an average value or a median value. The color information here is the color intensity value of the RGB component of the color.

  (Step S1705) The landmark image detection unit 305 acquires information indicating the combination of the colors of the landmark images in the n-th photographed image output in Step S1704, that is, information on two colors used as the landmark images. The color information described here may be values of color components such as RGB values, or may be information expressed by the above-described binarized color components. When there is only one color of the mark image, the color combination information may be information indicating that there is only one color and no other color.

  (Step S1706) The mark image detection unit 305 includes information indicating the color combination acquired in step S1705, the arrangement order of the pattern images stored in the image storage unit 301 prepared in advance in a memory, and the pattern image. The information indicating the order in which the pattern images in the captured image are arranged is acquired by comparing the information indicating the combination with the two colors used in the mark image. Even for a combination in which the mark image has only one color, the order in which the mark images are arranged may be set in advance so that information indicating the order in which the mark images are arranged can be acquired even when the mark image has only one color.

  (Step S1707) The mark image detection unit 305 binarizes the color information of each mark image in the m-th captured image. For example, the mark image detection unit 305 converts a combination of two colors of a mark image prepared in advance and converts these values into “1”, “0”, etc. when binarizing these colors. Is used to convert the color information of each landmark image into one of binary values “1” and “0”. Also, information indicating the order in which the pattern images in the captured image are arranged and whether the two colors included in the pattern image indicated by the arrangement order are converted to “1”, “0”, etc., respectively. The information on the color of each mark image may be converted into one of binary values “1” and “0”. The mark image detection unit 305 stores the binarized color information in a storage medium such as a memory in association with the position information of the mark image.

(Step S1708) The mark image detection unit 305 determines whether or not the counter m is greater than or equal to p. If it is greater than or equal to p, the process returns to the upper function, that is, the flowchart of FIG. If it is not greater than or equal to p, the process proceeds to step S1709.
(Step S1709) The mark image detection unit 305 increments the counter m by 1.
(Step S1710) The landmark image detection unit 305 reads out and acquires the m-th captured image from the captured image storage unit 304.

  (Step S1711) The mark image detection unit 305 indicates information indicating the positions of the plurality of mark images detected from the m = 1st photographed image in step S1703, and indicates the positions of the plurality of mark images in the mth photographed image. As in step S1704, information on the color of each mark image in the m-th photographed image is acquired and output in association with the position information of the mark image, for example, stored in a memory (not shown) or the like. Accumulate on media. Then, the process returns to step S1705.

  In the flowchart of FIG. 17, the information on the position of the mark image acquired from the captured image of m = 1 is used as the position information of the mark image of the captured image other than m = 1. This is limited to the case where the positions of the displays of the photographing unit 303 and the display unit 302 are fixed. In this way, instead of using the position information of the landmark image acquired from the captured image of m = 1 as the position information of the landmark image of the captured image other than m = 1, the same as step S1703 is performed for each captured image. Mark image detection processing may be performed, and finally mark images that match each captured image may be detected and associated using a technique such as pattern matching.

In order to make it easy to detect a mark image, a position image is prepared in advance, which has the same position as the other pattern images and a mark image with a color that is easy to detect. Only the position detection of the landmark image may be detected first using the captured image obtained by capturing the pattern image for use as the first captured image m = 1. A pattern image for position detection having a mark image of a color that is easy to detect is an image that is used only for detecting the position of the mark image, such as an image having a white mark image on a black background. An image having a strong contrast with the background.
In the flowchart of FIG. 17, the process ends when the power is turned off or the process ends.

Next, details of the process of detecting the correction parameter shown in step S1607 will be described using the flowchart of FIG.
(Step S1801) The correspondence information acquisition unit 306 substitutes 1 for a counter i.

  (Step S1802) The correspondence information acquisition unit 306 acquires the appearance pattern of the color of the landmark image at the position where the i-th landmark image is displayed in the captured image. Specifically, the i-th mark image in the captured image when a plurality of captured images are arranged in accordance with the information indicating the order in which the pattern images in the captured image obtained in step S1706 in FIG. 17 are arranged. For the position where is displayed, the appearance pattern of the color of the landmark image, here, as an example, the permutation in which the value obtained by binarizing the color of each landmark image appears. For example, the color at the position where the i-th landmark image in the captured image obtained by capturing the first pattern image is “0”, and the i-th landmark image in the captured image obtained by capturing the second pattern image. If the color at the displayed position is “1” and the color at the position at which the i-th mark image is displayed in the captured image obtained by photographing the third pattern image is “0”, the first to third colors For the captured image obtained by capturing the pattern image, the color appearance pattern at the position where the i-th landmark image acquired by the correspondence information acquisition unit 306 is displayed is “010”.

  (Step S1803) The correspondence information acquisition unit 306 is a correspondence management that indicates a correspondence relationship between the position of the landmark image in the plurality of original pattern images stored in the correspondence management information storage unit 307 and the color appearance pattern of the landmark image. Using the information, the position of the mark image in the pattern image corresponding to the color appearance pattern of the mark image at the position where the i-th mark image in the captured image is displayed is acquired. Specifically, the correspondence information acquisition unit 306 searches the correspondence management information for an appearance pattern that matches the appearance pattern of the color of the landmark image at the position where the i-th landmark image in the captured image is displayed. Information on the position of the mark image in the pattern image corresponding to the appearance pattern to be acquired is acquired.

(Step S1804) The correspondence information acquisition unit 306 includes information on the position where the i-th mark image in the photographed image is displayed, specifically the position information such as coordinate information, and the information stored in the image storage unit 301 acquired in step S1803. Correspondence information is obtained by associating the position where the landmark image stored in the pattern image is stored, specifically the position information such as coordinate information.
(Step S1805) The correspondence information acquisition unit 306 outputs the correspondence information acquired in step S1804, for example, stores it in a storage medium such as a memory.
(Step S1806) The correspondence information acquisition unit 306 increments the counter i by 1.
(Step S1807) The correspondence information acquisition unit 306 determines whether there is i-th position information with respect to the captured image. If there is i-th position information, the process returns to step S1802. If there is no i-th position information, the process returns to step S1808.

  (Step S1808) The color information acquisition unit 3081 responds to the color information of the mark information of the pattern information stored in the image storage unit 301 and the mark information of the pattern information in the shooting information obtained by photographing the pattern information. Information on the color of the mark information associated with the information is acquired. That is, the color information of the corresponding mark image is acquired from the pattern image and the captured image having the same arrangement order. The color information acquired here is, for example, the value of the color component. Here, only the color information of the pair of landmark information may be acquired, or the color information of the plurality of landmark information in the pattern image and the plurality of landmark information in the shooting information corresponding to the plurality of landmark information. Each of the color information may be acquired.

(Step S1809) The color correction information acquisition unit 3082 is a function for correcting the color information of the pattern image in the photographed image to the original color information of the pattern image using the color information acquired in Step S1808. Color correction information which is information such as a coefficient of a certain color correction function is calculated. Note that in step S1808, when color information of a plurality of positions is acquired, the color correction information may be calculated using a least square method or the like.
(Step S1810) The color correction information acquisition unit 3082 accumulates the color correction information calculated in step S1809 in the correction parameter storage unit.

(Step S1811) The position correction information acquisition unit 3083 uses the correspondence information acquired in step S1804, and shows a function indicating the relationship between the position information of the pattern image in the captured image and the information of the original position of the pattern image. Position correction information which is information such as a coefficient of the position correction function is calculated. A known position correction function can be used as the position correction function.
(Step S1812) The position correction information acquisition unit 3083 accumulates the position correction information calculated in step S1811 in the correction parameter storage unit. Then, the process ends.
In the flowchart of FIG. 18, the process is terminated by powering off or a process termination interrupt.

Hereinafter, a specific operation of the image processing apparatus according to the present embodiment will be described. A conceptual diagram of the image processing apparatus 300 is shown in FIG. Here, as an example, a case will be described in which the image processing apparatus 300 is realized by a computer 300a other than the display 302a and the photographing unit 303 which are display devices of the display unit 302. However, hardware other than the display 302a and the photographing unit 303 may be realized. For example, it is assumed that this image processing apparatus 300 constitutes a video conference terminal as in FIG. 5 of the first embodiment.
Here, it is assumed that the mark image in each pattern image has one of two colors. The shape of the mark image is assumed to be circular. Here, as an example, it is assumed that the size of the pattern image is the same as the size of other images normally displayed on the display unit 302, for example, the size of an image such as a video conference.

  FIG. 20 is a pattern image color management table for managing the color combinations of the mark images for each pattern image. The pattern image color management table has attributes of “pattern ID”, “0: color”, and “1: color”. “Pattern ID” is identification information of a corresponding pattern image. This pattern ID also indicates the order of arrangement of pattern images. “0: Color” indicates a color that is determined to be “0” during binarization. “1: Color” indicates a color determined to be “0” in binarization. “0: Color” and “1: Color” indicate two colors of the mark image of the pattern image. Here, as an example, the color of the mark image is represented by a combination of RGB components, and the RGB components are represented by binary values “0” or “1”, respectively.

  For example, if the R, G, and B color components of the landmark image are actually represented by 8 bits, when the R, G, and B color component values are 256, the respective components are expressed as “ When it is represented by “1” and less than 256, each component is represented by “0”. That is, “0: color” and “1: color” are represented by an array of R values, G components, B components, and binary values, respectively. For example, red is represented by (100), blue is represented by (010), and green is represented by (001). Here, each color component is binarized with “256” as a threshold value, but the threshold value may not be “256”.

In FIG. 20, for example, two color (001) and (010) are used as a mark image in a pattern image whose “pattern ID” is “1”. Here, it is assumed that the area other than the mark image of the pattern image, that is, the color of the background image is black (000).
In each pattern image, A (A is an integer of 2 or more) mark images are arranged so as not to overlap each other, and A mark images are arranged at the same position between the pattern images. It is assumed that Further, when the pattern images are arranged in a preset order, the appearance pattern of the color of the mark image appearing at each position where the mark image is arranged in the pattern image is different from each other for each position. To do.

  That is, in order to satisfy such a condition, when the number of pattern images is B (B is an integer of 2 or more),

2 ^B ≧ A

  To satisfy the relationship. In this specific example, a case will be described in which B = 6 pattern images in which A = 64 mark images are arranged are prepared. Here, it is assumed that the mark images are arranged at intervals of 100 pixels, for example. Further, it is assumed that the diameter of each mark image is 10 pixels, for example.

  FIG. 21 schematically shows six pattern images used here. Here, the color of the mark image of each pattern image is represented by a binarized color using the pattern image color management table shown in FIG. That is, for example, in FIG. 21, for the pattern image whose “pattern ID” is “2”, the color of (001) is replaced with “0” and the color of (010) is replaced with “1”. In the pattern image shown in FIG. 21, when the pattern images are arranged in ascending order of the “pattern ID” values, for example, the color appearance pattern at the position where the mark image in the upper right corner is arranged is “000111”. .

  The position where each mark image of each pattern image is arranged is represented by coordinate information in the pattern image. Here, when the pattern images are arranged in ascending order of the “pattern ID” values as shown in FIG. 21, the information indicating the relationship between the information on the position where the mark image is arranged and the appearance pattern of the color of the mark image The correspondence management information is shown in FIG. The correspondence management information has attributes of “coordinates” and “appearance pattern”. The “coordinate” is a position where the mark image in each pattern image is displayed, and here indicates coordinate information of the center of the mark image. The “appearance pattern” indicates a color appearance pattern binarized for each pattern image.

First, when an instruction to acquire a correction parameter is given to the image processing apparatus 300 via a reception unit (not shown) or the like, the display unit 302 displays a pattern image to be displayed first among the pattern images in the image storage unit 301. Is read out and displayed on the display 302a.
FIG. 23 is a diagram illustrating a display example of the first pattern image displayed on the display 302a. Here, it is assumed that a pattern image whose “pattern ID” is “3” is displayed as the first pattern image. The landmark image 231 is a landmark image whose color is a color (001) corresponding to the “0: color” attribute, and the landmark image 232 is a landmark image whose color is a color (100) corresponding to the “1: color” attribute. Suppose that

  The imaging unit 303 captures the pattern image displayed on the display 302a by the display unit 302 and captures the captured image. A captured image that is a captured image is received by the image receiving unit 104 and accumulated in the captured image storage unit 304. Here, it is assumed that an image captured and accumulated is one frame of a moving image. The display time of each pattern image is not limited as long as the photographing unit 303 can complete photographing.

FIG. 24 is a diagram illustrating a captured image obtained by capturing the pattern image displayed on the display 302 a as illustrated in FIG. 23 captured by the capturing unit 303.
When the display and shooting of the first pattern image are completed, the remaining pattern images are also sequentially displayed and shot. The display order of the pattern images displayed on the display unit 302 and the order of the “pattern ID” described above do not have to match. Here, the display of the pattern image is started with the power-on as a trigger, but the display start trigger or the like is not limited.

  Next, the landmark image detection unit 305 reads one of the photographed images as shown in FIG. 24, here, the photographed image obtained by photographing the pattern image displayed at the first, and performs a process of detecting the landmark image. Here, since it is known that the color used for the mark image is any one of (001), (011), (100), (110), (101), and (111), For each pixel, it is sequentially determined whether or not it corresponds to any one of the above colors, and a region adjacent to a pixel having a color corresponding to any one of the above colors is detected.

  In order to prevent erroneous detection of the mark image due to the color of the part other than the display area of the display 302a, the display area of the display 302a is detected in advance by detecting the frame or background of the display, or the display area. The mark image may be detected only in the display area by accepting the designation about the image via an input device or the like. However, since the photographed image is an image obtained by temporarily displaying the pattern image on the display 302a by the photographing unit 303, the color of the mark image of the pattern image in the photographed image is different from the color of the original pattern image. It is conceivable that the component values are different. For this reason, here, a threshold value for each color component in consideration of such a color change is designated, and the color component of the pixel in the photographed image is binarized by this threshold value, and the color of the mark image described above is used. Judge whether there is.

  For example, the respective threshold values of the R, G, and B components are set to “180”, and “1” is set to “180” or more and “0” is set to less than “180” for each color component of each pixel. To determine whether the pixel matches any of the above colors. For example, when the R, G, and B components are 13, 40, and 150, respectively, the color of the pixel is converted to (000) and does not match any of the above colors. It is judged. For example, if the R, G, and B components are 230, 80, and 192, respectively, the color of the pixel is converted to (101) and matches one of the above colors, so that a landmark image is formed. It is determined to be a pixel. Then, the areas adjacent to the pixels determined to be the same color are determined as mark image areas, and the coordinates of the center pixel of the area and the color value of the mark image are stored in a storage medium such as a memory (not shown). accumulate. Here, the coordinates of the pixel at the center of the mark image are set to the coordinates of the pixel at the position where the diameter of the mark image in the x-axis direction is the longest and the diameter in the y-axis direction is the longest. Is done.

  Furthermore, if two colors that can be taken by the mark image are detected while detecting the mark image for one photographed image, the two colors are managed by the pattern image color management table shown in FIG. The attribute value of the “pattern ID” of the matching record is acquired by comparing with the combination of existing colors, that is, the combination of “0: color” and “1: color”. That is, the photographed image is determined to be an image obtained by photographing the pattern image associated with the acquired “pattern ID” attribute value. For example, when the colors of the two mark images are (001) and (100), “3” is acquired as the value of “pattern ID” from the pattern image color management table of FIG. Then, the acquired attribute value of “pattern ID” is stored in association with information indicating the positions and the like of all the landmark images detected from the same captured image. Further, the landmark image detection unit 305 binarizes the two colors of the landmark image using the pattern image color management table of FIG.

  Specifically, here, the colors of the two landmark images are (001) and (100), and the record includes a combination of the two colors, that is, the record whose attribute value of “pattern ID” is “3”. Since (001) is a “0: color” attribute and (100) is a “1: color” attribute, (001) is converted to “0” and (100) is changed to “1”. Converted. The binarized color values are also stored in association with information indicating the positions and the like of all the landmark images detected from the same photographed image.

  Next, the same processing as described above is repeated for the other five photographed images, in this case, the photographed images obtained by photographing the pattern images except for “pattern ID” “3”. However, here, since the shooting direction, shooting area, and the like of the shooting unit 303 are fixed when the pattern image is shot, the process of detecting the mark image is omitted, and the mark detected for the first shot image is detected. Information indicating the area of the image is directly used as information indicating the area of the mark image of another captured image. Therefore, the coordinates indicating the position of the mark image at the same position in different captured images have the same value.

  FIG. 25 is a mark image management table for managing the position and color of the mark image detected from the photographed image. In the mark image management table, they are “pattern ID”, “coordinate”, “color”, and “binarized color”. “Pattern ID” is a “pattern ID” obtained by using a combination of two colors used in a landmark image detected in one captured image. The “coordinate” is a value indicating the position of the mark image with the center coordinates of the mark image. Here, the coordinates are represented by xy coordinate values in pixel units, where the upper left corner of the image is (0, 0). “Color” is a value represented by a color component obtained by binarizing the color of the mark image. The “binarized color” is a value obtained by binarizing the color of the mark image in accordance with the photographed pattern image based on the pattern image color management table shown in FIG. Note that the value of “coordinate” is a value for explanation, and is different from an actual value or the like.

  Next, the correspondence information acquisition unit 306 binarizes the color of the mark image at each position where the mark image detected by the mark image detection unit 305 is arranged, that is, “binary color” in FIG. An array pattern in which the attribute values are arranged in ascending order of the corresponding “pattern ID” attribute is acquired. This arrangement pattern is a color appearance pattern at a position where each mark image is arranged in the photographed image. The correspondence information acquisition unit 306 temporarily stores the acquired appearance pattern in a memory or the like in association with the position of the mark image, here the coordinates. For example, the appearance pattern obtained at the position of the coordinates (120, 140) is “010010”.

  FIG. 26 is an appearance pattern management table for managing the position of the mark image and the color appearance pattern in the captured image temporarily stored in the memory or the like. The appearance pattern management table has attributes of “marking position ID”, “coordinates”, and “shooting appearance pattern”. “Mark position ID” is identification information for managing the position where each mark image is arranged, “Coordinate” is the coordinate of the position where the mark image is arranged, here the coordinates of the center pixel of the mark image, “ The “photographing appearance pattern” is a pattern in which binarized color values are arranged in ascending order of the “pattern ID” attribute.

Here, it is assumed that correspondence management information as illustrated in FIG. 22 is stored in the correspondence management information storage unit 307 in advance. This correspondence management information may be set by the user, or obtained from the pattern image by the same process as described above using the landmark image detection unit 305 and the correspondence information acquisition unit 306 as described above. May be the information.
The correspondence information acquisition unit 306 shows, for each record in the appearance pattern management table as shown in FIG. 26, a record having an attribute value of “appearance pattern” that matches the attribute value of “shooting appearance pattern” in FIG. Search in correspondence management information. Then, the “coordinate” attribute value of the matching record is associated with the “coordinate” attribute value of the appearance pattern management table. Then, the correspondence information obtained by the association is stored in a storage medium such as a memory (not shown) by the correspondence information acquisition unit 306.

  FIG. 27 is a diagram illustrating correspondence information output by the output unit 109. The correspondence information has attributes of “original coordinates” and “in-photographed image coordinates”. “Original coordinates” are coordinates at which the mark image in the original, that is, the pattern image stored in the image storage unit 301 is arranged. “Coordinates in photographed image” is coordinates indicating the position where the mark image arranged at “original coordinates” is arranged in the photographed image. “Original coordinates” and “coordinates in captured image” of the same record correspond to each other. That is, it can be considered that the portion of the captured image where the pattern image is displayed is an image obtained by converting the coordinates indicated by the “original coordinates” into the coordinates indicated by the “coordinates in the captured image”.

  Next, the color information acquisition unit 3081 acquires the color information of the coordinates of the corresponding mark image indicated by the corresponding information from one of the pattern images and the captured image obtained by capturing the pattern image. Specifically, the color information of the pixel at the coordinate indicated by the “original coordinate” of each record of the correspondence information is acquired from the pattern image, and the “coordinate in the captured image” of the same record as the “original coordinate” is acquired from the captured image. The color information of the pixel at the coordinates indicated by is acquired. Since the color information acquired here is information used for color correction, each component value of the R, G, and B components, when the color is expressed in 24-bit color, for example, an 8-bit value, Need to be. Then, the color correction information acquisition unit 3082 corrects the color of the captured image to the color of the original pattern image by using the color of the corresponding coordinates respectively acquired by the color information acquisition unit 3081 from the pattern image and the captured image. The color correction information that is a parameter of the color correction function that is a function for correcting the color is calculated.

  For example, such a color correction function is assumed to be the same as the color correction information described in the specific example in the first embodiment. That is, if the pixel value of the captured image is (r, g, b) and the corresponding pixel value of the original pattern image is (R, G, B),

R = a ₁ r + a ₂ g + a ₃ b,
G = b ₁ r + a ₂ g + a ₃ b,
B = c ₁ r + a ₂ g + a ₃ b

It is represented by Note that a _{1 to} a ₃ , b _{1 to} b, and c _{1 to} c ₃ are coefficients of a color correction function, that is, color correction information.

  By substituting the color information of the corresponding pixels of the pattern image and the captured image acquired by the color information acquisition unit 3081 for this color correction function, the value of the coefficient is obtained, whereby the color of the captured image is changed to the original color. A color correction function for correcting the color of the pattern image can be obtained. The color correction information acquisition unit 3082 accumulates the acquired coefficient of the color correction function in the correction parameter storage unit 106 as color correction information. Here, it is preferable to use the least square method or the like when calculating the above-described coefficient for the color information acquisition unit 3081 to acquire color information of corresponding pixels at four or more locations.

  However, in order to obtain the color correction function shown here, the color information of the corresponding pixels at three different positions is usually required. By using this color correction function, the color of the entire captured image can be corrected to the same color as the original pattern image. That is, it is possible to match the colors of the image displayed on the display unit 302 and the image displayed on the display unit in the image captured by the imaging unit 303. Here, the case where the function for correcting the color of the captured image to the color of the pattern image has been described as the color correction function. However, depending on the application, a function for correcting the color of the pattern shadow image to the color of the captured image may be used. You may make it use.

  Next, the position correction information acquisition unit 3083 acquires position correction information that is a parameter of a position correction function that is a function for correcting the position, using the correspondence information. For example, it is assumed that the position correction function used here is the same as the position correction information described in the specific example in the first embodiment. That is, if the pixel position in the captured image is (x, y) and the pixel position in the original pattern image is (X, Y), the position correction function is

X = ax + by,
Y = cx + dy

  It is represented by Note that a, b, c, and d are coefficients of a position correction function, that is, position correction information.

  By substituting the coordinates of the pattern image indicated by the correspondence information and the coordinates in the captured image corresponding to this position correction function, the position of the pixels constituting the captured image is determined. A position correction function for converting the coordinates to the position of the pixel in the pattern image can be obtained. The position correction information acquisition unit 3083 accumulates the acquired coefficient of the position correction function in the correction parameter storage unit 106 as position correction information.

  In this case, since the correspondence information includes information on the coordinates of the corresponding pixels at three or more locations, it is preferable to use the least square method or the like when calculating the above coefficients. However, in order to obtain the position correction function shown here, information on the positions of the corresponding pixels at two different positions is usually required. By using this position correction function, it is possible to correct the pixel position of the captured image, the shape of the image, and the like to the same position, shape, and the like as the original pattern image.

FIG. 28 is a diagram showing a correction parameter management table configured by the correction parameter configuration unit 308 and managing the correction parameters stored in the correction parameter storage unit 106. In the correction parameter management table, “color correction information” indicates color correction information configured by the color correction information acquisition unit 3082, and “position correction information” indicates position correction information configured by the position correction information acquisition unit 3083. Show. Α _{1 to} α ₃ , β _{1 to} β ₃ , and γ _{1 to} γ ₃ are values corresponding to the coefficients a _{1 to} a ₃ , b _{1 to} b, and c _{1 to} c ₃ of the color correction function exemplified above. Suppose that A, B, C, and D are values corresponding to the coefficients a, b, c, and d of the position correction function exemplified above, respectively.

Using the correction parameters stored in the correction parameter storage unit 106 as described above, the image processing apparatus 300 changes from the second image captured by the imaging unit 303 to the first image, as in the first embodiment. A matching image area is detected, and a second image is output with this area as the transmission area. Since this process is the same as that of the first embodiment, description thereof is omitted.
Here, the case where the function for correcting the position of the pixel in the captured image to the position in the pattern image has been described as the position correction function. However, the position of the pixel in the pattern image in the captured image can be changed as necessary. A function for correcting the position may be used.

Note that the color correction function and position correction function described in this specific example are merely examples, and other color correction functions, position correction functions, and the like may be used. In such a case, according to these functions. The correction parameter may be calculated using correspondence information or the like. Also,
In the above specific example, in the detection process of the mark image, the color of the mark image is compared with the color of each pixel. However, the mark image may be detected by other methods. For example, when it is assumed that the shape of the mark image does not change significantly, such as when the image quality of the pattern image in the captured image is good, an area having almost the same shape as the mark image is generally used. The mark image may be detected by detecting it with a simple pattern matching technique or the like.

  In the above specific example, the case where each of the plurality of landmark images of each pattern image has any one of two colors has been described. However, in the present embodiment, a plurality of landmark images are used. May have any one of two or more colors that can be identified in the captured image. In this case, when the number of colors of the mark images in each pattern image is C (C is an integer of 2 or more), the number of mark images is A, and the number of pattern images is B (B is a positive integer),

C ^B ≧ A

  Should be satisfied.

  As described above, according to the present embodiment, by displaying the pattern image, between the image displayed on the display unit 302 and the image displayed on the display unit 302 in the image shot by the shooting unit 303. By calculating correspondence information that can accurately correspond to the position and obtaining the correction parameter using the correspondence information, the first image and the second region shown in the first embodiment are obtained. In the process of detecting the matching region, it is possible to appropriately correct the color and position of the first image and the second region image, and the matching region between the first image and the second image. Can be accurately detected. Thereby, the area | region where the 1st image in the 2nd image is reflected can be correctly designated as a transmissive area | region.

  Further, in the present embodiment, the landmark image is identified and detected from within the photographed image by the appearance pattern of the color of the landmark image. Therefore, only a part of the display unit 302 is detected depending on the photographing situation of the photographing unit 303. Even in the case where it is impossible to photograph and it is not known which part in the display area of the display unit 302 has been photographed, or when the photographing direction or the like of the display unit 302 cannot be determined, the placement of the landmark image is accurately performed from within the photographed image. Can be identified.

Further, in the present embodiment, the landmark image is identified by the appearance pattern of the color of the landmark image in the photographed image. Therefore, even if the color of the landmark image is slightly discolored by processing such as display or photographing, the photographed image The landmark image can be accurately identified from the inside.
Also, unlike the case of identifying the landmark image from the shape of the landmark image, the landmark image is identified by the appearance pattern of the color, so that the photographed image is distorted or a large perspective is included depending on the photographing position. Even if the mark image is deformed due to distortion, the mark image can be accurately identified from the captured image.
In the present embodiment, information for correcting the position and color can be obtained simply by photographing a pattern image prepared in advance by the photographing unit 303 actually used for photographing. It is not necessary to install a dedicated device or the like for the cost, and an increase in cost can be suppressed. In addition, since a special operation for correction can be minimized, the operability is improved.

  In the above embodiment, the position where the mark image is detected is identified by the appearance pattern of the color of the mark image. However, at the position where the mark image is arranged in at least one pattern image of the plurality of pattern images. The position where the mark image is arranged may be identified by the appearance pattern that is the pattern of the presence or absence of the mark image. For example, a mark image appears at a certain coordinate (x1, y1) of the first pattern image, and a mark image does not appear and only a background image appears in different second and third patterns at the same coordinates. In this case, the mark image may be replaced with “1” and the background image may be replaced with “0”, and the appearance pattern of the mark image at the coordinates (x1, y1) may be considered as “011”. However, in this case, since the position of the landmark image cannot be detected until the landmark image is detected in a plurality of captured images, once the position of the landmark image is detected, the background image is detected at the same position of the other captured images. It is necessary to detect, and there is a possibility that the process may be reversed.

  Further, in the above embodiment, a plurality of pattern images each having one mark image are respectively displayed at predetermined positions different from each other, and each of the pattern images in the plurality of captured images is displayed. By detecting the position of each mark image, the combination of the captured pattern image and the detected position of the mark image in the captured image is used to identify the detected position of the captured mark image. It may be. That is, here, the position of the mark image is specified for each pattern image. Therefore, it is possible to identify the position where the mark image of the photographed image is detected by determining which pattern image is detected from the photographed image obtained by photographing each landmark image.

  In other words, here, the information for identifying the pattern image in which each landmark image is detected is regarded as the appearance pattern for each landmark image, and the appearance pattern for each landmark image detected from each captured image is acquired. Using the appearance pattern, the position of the landmark image detected in each captured image and the position where the landmark image is arranged in the plurality of original pattern images having the appearance pattern that matches the appearance pattern of the landmark image The correspondence information indicating the correspondence is acquired. For example, in the first pattern image, when only one mark image is arranged at the coordinates (x2, y2), the position where the mark image is detected in the captured image obtained by shooting the first pattern image is It can be determined that the position corresponds to the coordinates (x2, y2) of the first pattern image.

  However, in such a case, for example, which pattern image the pattern image captured by the captured image is associated with information for identifying the captured pattern image, or the display order and the capture order of the pattern images Need to be identified by associating them with each other. In such a case, since only one mark image can be arranged in each pattern image, in order to identify a plurality of positions in the captured image, the number of positions to be identified is determined. It is necessary to prepare the same number as the number of images and to display all these pattern images. For this reason, it is conceivable that the processing time becomes longer by the appearance pattern of the color of the mark image as described above as compared with the case of identifying a plurality of positions where the mark image is arranged.

In the first and second embodiments, each processing (each function) may be realized by centralized processing by a single device (system), or distributed processing by a plurality of devices. May be realized.
In the first and second embodiments, each component may be configured by dedicated hardware, or a component that can be realized by software may be realized by executing a program. . For example, each component can be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

In the software program for realizing the image processing apparatus according to each of the first and second embodiments, in the transmission step for transmitting information, the reception step for receiving information, etc., processing performed by hardware, for example, the transmission step This does not include processing performed by a modem or an interface card (processing performed only by hardware).
Further, this program may be executed by being downloaded from a server or the like, and a program recorded on a predetermined recording medium (for example, an optical disk such as a CD-ROM, a magnetic disk, a semiconductor memory, or the like) is read out. May be executed by
Further, the computer that executes this program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.
Further, in each of the first and second embodiments, it goes without saying that two or more communication means (such as an information transmission unit) existing in one apparatus may be physically realized by one medium.

The present invention is not limited to the above-described embodiments, and various modifications are possible, and it goes without saying that these are also included in the scope of the present invention.
Further, although cases have been described with the above embodiments where the image processing apparatus is a stand-alone, the image processing apparatus may be a stand-alone apparatus or a server apparatus in a server / client system. In the latter case, the output unit or the reception unit receives an input or outputs a screen via a communication line.

  As described above, the image processing apparatus or the like according to the present invention is suitable as an image processing apparatus or the like that processes an image obtained by photographing an image displayed on a display or the like, and is used as an image processing apparatus or the like used for a video conference or the like. Useful.

1 is a block diagram of an image processing device in Embodiment 1. FIG. 6 is a flowchart for explaining the operation of the image processing apparatus according to the embodiment. 6 is a flowchart for explaining the operation of the image processing apparatus according to the embodiment. It is a block diagram which shows the structure of the video conference system using the image processing apparatus in the embodiment. It is a notional block diagram of the video conference system using the image processing apparatus in the embodiment. It is a typical block diagram which shows the relationship between the 2nd image in the same embodiment, and the 1st image image | photographed by the 2nd image. It is a block diagram which shows the example of a display of the 1st image in the embodiment. It is a block diagram which shows the example of a display of the 2nd image in the embodiment. It is a block diagram which shows the area | region which corresponds in the 2nd image in the embodiment. It is a block diagram for demonstrating the alpha channel in the embodiment. It is a block diagram which shows the 2nd image clipped in the embodiment. It is a block diagram which shows the clipping image management table | surface in the embodiment. It is a block diagram which shows the example of a display of the 2nd image in the 2nd video conference terminal in the embodiment. It is a block diagram which shows the modification of the 1st video conference terminal in the embodiment. 6 is a block configuration diagram of an image processing apparatus according to Embodiment 2. FIG. 6 is a flowchart for explaining the operation of the image processing apparatus according to the embodiment. 6 is a flowchart for explaining the operation of the image processing apparatus according to the embodiment. 6 is a flowchart for explaining the operation of the image processing apparatus according to the embodiment. It is a notional block diagram of the image processing apparatus in the embodiment. It is a block diagram which shows the pattern image color management table | surface in the embodiment. It is a block diagram which shows the example of the pattern image in the same embodiment. It is a block diagram which shows the correspondence management information in the embodiment. It is a block diagram which shows the example of a display of the information processing apparatus in the embodiment. It is a block diagram which shows the example of the picked-up image of the information processing apparatus in the embodiment. It is a block diagram which shows the mark image management table | surface in the embodiment. It is a block diagram which shows the appearance pattern management table | surface in the embodiment. It is a block diagram which shows the correspondence information in the same embodiment. It is a block diagram which shows the correction parameter management table | surface in the embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 1st video conference terminal 20 2nd video conference terminal 61, 70 1st image 62, 80, 91, 92 2nd image 90 Matching area 95 Alpha channel 100, 300 Image processing apparatus 101 Reception part 102a, 302a Display 102, 302 Display unit 103, 303 Imaging unit 104 Image reception unit 105 Delay buffer 106 Correction parameter storage unit 107 Comparison unit 108 Transparent image configuration unit 109 Output unit 201 Imaging unit 202 Transmission unit 203 Reception unit 204a Display 204 Display unit 231, 232 Mark image 300a Computer 301 Image storage unit 304 Captured image storage unit 305 Mark image detection unit 306 Corresponding information acquisition unit 307 Corresponding management information storage unit 308 Correction parameter configuration unit 1000, 2000 Information processing device 3081 Color information Acquiring means 3082 color correction information acquisition unit 3083 position correction information acquisition means

Claims (16)

A reception unit for receiving the first image;
A display unit for displaying the first image received by the reception unit;
An image receiving unit that receives a second image that is an image of an area including the first image displayed by the display unit;
A correction parameter storage unit for storing correction parameters used for correction for color matching and alignment between the first image and the first image included in the second image;
Based on the correction parameter, the first image or the image in the region where the first image in the second image is located is corrected, and the first image and the second image A comparison unit that compares the image in the region where the first image is located;
Using the comparison result of the comparison unit, a transmission image configuration unit that generates a transmission image in which a region that matches the first image in the second image is set as a transmission region;
An image processing apparatus comprising: an output unit that outputs a transparent image generated by the transparent image forming unit. The comparison unit includes:
The image processing apparatus according to claim 1, wherein the image in the region where the first image and the first image in the second image are located is corrected based on the correction parameter. The transmission image constructing unit includes:
The image processing apparatus according to claim 1, wherein a plurality of transmission images are formed by cutting out images of an area separated by the transmission area in the second image. The transmission image constructing unit includes:
The image processing apparatus according to claim 1, wherein a transmission image having an alpha channel that designates a transmission area is generated. An image storage unit that stores a plurality of pattern images in which mark images, which are images serving as marks, are arranged at different positions, and in which a plurality of pattern images having different appearance patterns appear between the plurality of pattern images. When,
A landmark image detection unit for detecting the landmark image from within the pattern photographed image;
Using each mark image detected by the mark image detection unit, an appearance pattern for each mark image is acquired, and the position of the mark image detected in each pattern photographed image, and the appearance pattern of the mark image A correspondence information acquisition unit that acquires correspondence information indicating correspondence with positions where mark images are arranged in the plurality of pattern images having matching appearance patterns;
The correction parameter is configured using the correspondence information acquired by the correspondence information acquisition unit, and further includes a correction parameter configuration unit that accumulates in the correction parameter storage unit,
The display unit
Switch and display a plurality of pattern images stored in the image storage unit,
The image receiving unit receives a pattern captured image that is an image obtained by capturing an image including a plurality of pattern images displayed by the display unit,
5. The image processing apparatus according to claim 1, wherein the mark image detection unit detects the mark image from a pattern photographed image received by the image receiving unit. The image storage unit
The mark image, which is a mark image having any one of a plurality of colors, is a plurality of the pattern images arranged in different positions, and the pattern images are between the pattern images. The plurality of pattern images are stored at the same position, and the appearance images of the colors of the mark images arranged at different positions are different from each other.
The correspondence information acquisition unit
Using each mark image detected by the mark image detection unit, a color appearance pattern for the mark image detected at the same position in each pattern image is acquired, and the color appearance pattern is acquired Obtaining correspondence information indicating a correspondence between a position in the photographed image and a position where the mark image is arranged in the plurality of pattern images having the appearance pattern of the color that matches the appearance pattern of the color. The image processing apparatus according to claim 5. A correspondence management information storage unit that stores correspondence management information that is information indicating correspondence between positions where the plurality of mark images in the plurality of pattern images are arranged and color appearance patterns at the arranged positions; And
The correspondence information acquisition unit
The image processing apparatus according to claim 6, wherein the correspondence information is acquired from a color appearance pattern acquired for a landmark image in the captured image using the correspondence management information. The image storage unit
A plurality of patterns satisfying the relationship of 2 ^B ≧ A, where A (A is an integer of 2 or more) and B (B is an integer of 2 or more) of the pattern images in the pattern images. The image processing apparatus according to claim 6, wherein an image is stored. Mark images of the plurality of pattern images are different from each other for each pattern image,
The landmark image detection unit includes:
Obtain the landmark image detected from each pattern image,
The correspondence information acquisition unit
The mark image detection unit acquires the mark image according to the order in which the pattern images are acquired, and acquires the correspondence information. 9. Image processing apparatus. The correction parameter configuration unit includes:
Color information that is information relating to the color of the target image in the pattern photographed image received by the image receiving unit, and the color of the target image in the pattern image corresponding to the target image indicated by the correspondence information output by the output unit Color information acquisition means for acquiring information;
Using the color information acquired by the color information acquisition means, acquires color correction information, which is a correction parameter used for correction for color matching of the image received by the image receiving unit, and accumulates it in the correction parameter storage unit The image processing apparatus according to claim 5, further comprising a color correction information acquisition unit that performs the correction. The correction parameter configuration unit includes:
Using the position information acquired by the correspondence information acquisition unit, position correction information, which is a correction parameter used for correction for image alignment received by the image receiving unit, is acquired and stored in the correction parameter storage unit The image processing apparatus according to claim 5, further comprising position correction information acquisition means for performing the correction. A reception unit that receives a first image, a display unit that displays an image, an image reception unit that receives a second image, the first image, and a first image included in the second image; A correction parameter storage unit that stores correction parameters used for correction for color matching and registration, a comparison unit that compares images, a transmission image configuration unit that generates a transmission image, and an output unit An image processing method in an image processing apparatus comprising:
The accepting unit accepting the first image;
The display unit displaying the received first image;
The image receiving unit receiving a second image that is an image of a region including the displayed first image;
The comparison unit correcting the first image or the image in the region where the first image in the second image is located based on the correction parameter stored in the correction parameter storage unit; ,
The comparing unit comparing the first image with an image in a region where the first image in the second image is located;
The transparent image constructing unit, using the comparison result, generating a transparent image in which a region that matches the first image in the second image is set as a transparent region;
And a step of outputting the generated transmission image by the output unit. The image processing apparatus further includes a plurality of pattern images in which the mark images, which are images serving as marks, are arranged at different positions, and a plurality of patterns in which the appearance patterns of the mark images are different between the plurality of pattern images. An image storage unit for storing an image, a landmark image detection unit for detecting the landmark image from the pattern photographed image, a correspondence information acquisition unit, and a correction parameter configuration for generating correction parameters to be accumulated in the correction parameter storage unit And comprising
The display unit switching and displaying a plurality of pattern images stored in the image storage unit;
The image receiving unit receiving a pattern captured image that is an image obtained by capturing an image including a plurality of pattern images displayed by the display unit;
The landmark image detection unit detecting the landmark image from within a pattern photographed image received by the image reception unit;
The correspondence information acquisition unit acquires an appearance pattern for each landmark image using each landmark image detected by the landmark image detection unit, and the position of the landmark image detected in each pattern photographed image, Obtaining correspondence information indicating correspondence with positions where the mark images are arranged in the plurality of pattern images having an appearance pattern that matches the appearance pattern of the mark image;
The correction parameter configuration unit configures the correction parameter using the correspondence information acquired by the correspondence information acquisition unit, and accumulates the correction parameter in the correction parameter storage unit;
The image processing method according to claim 12, further comprising: In the image storage unit,
The mark image, which is a mark image having any one of a plurality of colors, is a plurality of the pattern images arranged in different positions, and the pattern images are between the pattern images. The plurality of pattern images are stored at the same position where the plurality of mark images are arranged, and the appearance patterns of the colors of the mark images arranged at different positions are different from each other.
The correspondence information acquisition unit
Using each mark image detected by the mark image detection unit, a color appearance pattern for the mark image detected at the same position in each pattern image is acquired, and the color appearance pattern is acquired Obtaining correspondence information indicating a correspondence between a position in the captured image and a position where a mark image is arranged in the plurality of pattern images having an appearance pattern of a color that matches the appearance pattern of the color;
The image processing method according to claim 13, further comprising: A computer having a correction parameter storage unit that stores correction parameters used for correction for color matching and alignment between the first image and the first image included in the second image,
Receiving the first image;
Displaying the received first image;
Receiving a second image that is an image of an area including the displayed first image;
Correcting the first image or an image in a region where the first image in the second image is located based on the correction parameter stored in the correction parameter storage unit;
Comparing the first image with an image in a region where the first image in the second image is located;
Using the comparison result, generating a transmission image in which a region in the second image that matches the first image is set as a transmission region;
A program for executing the step of outputting the generated transparent image.   A computer-readable recording medium on which the program according to claim 15 is recorded.

Images