JP5445549B2 - Image communication apparatus, image communication method, program, and image communication system - Google Patents

Description

  The present invention relates to an image communication apparatus, an image communication method, a program, and an image communication system.

  In recent years, with the spread of broadband and cloud environments, and the development of communication tools accompanying the development of information and communication technologies, working styles (work styles) that are not constrained by location and time are spreading. As a result, it is predicted that the main form of collaborative work in the group will also change from a general single-concentration / same-space type to a remote distributed cooperative type. In order to realize this, a unified communication (registered trademark) technology / system that integrates voice communication, FAX, electronic mail, instant messenger, VoIP, and the like has been attracting attention.

  As rich information transmitted by the unified communication system, video and sound information occupy an important position. For example, a tool based on a conventional video conference system is installed as a position function of many unified communication systems.

  By the way, the so-called TV conference system is a communication tool mainly intended for formal communication that is intentional and formal / official, such as discussion after agenda has been decided. On the other hand, in recent years, it is important to nurture a sense of unity and human relations in order to smoothly promote team collaboration, and to enhance the sense of unity and human relationships. It is pointed out that unplanned and accidental informal communication is important.

  As a tool of the unified communication system to support such informal communication, it is not a planned form such as “who is in the A meeting room from what time to what time” as in the TV conference, but is unplanned. It is conceivable to share daily scenes of offices, that is, to constantly communicate and present environmental video and sound information between remote offices.

  In addition, regarding the above-described unified communication system, for example, Patent Document 1 discloses a technique for preventing background images from being transmitted by not transmitting video images captured by an imaging apparatus as they are, but by not transmitting video portions with small motion. Is disclosed. Japanese Patent Application Laid-Open No. 2004-228561 discloses a technique for providing only a part of a video area to a person who has been confirmed to have a proper access authority.

JP-A-5-91407 JP 2001-186501 A

  However, the unified communication system for supporting the above informal communication has a serious privacy problem. Hereinafter, this point will be described.

  The in-office video that is always taken unplanned can be viewed unilaterally from the remote office side of the distribution destination. In the same space, if there is a person staring at him / herself in the field of view, the person being viewed will notice that person. However, for example, when an in-office image captured by an imaging device installed in the corner of a room is distributed to a remote office, the person included in the in-office image determines who is watching or who is watching It is difficult to distinguish. For this reason, a secret unilateral remote browsing action that can lead to privacy problems may occur.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to provide a new and improved image communication apparatus and image communication for supporting remote communication while protecting privacy. A method, a program, and an image communication system are provided.

  In order to solve the above-described problem, according to one aspect of the present invention, a first captured image is acquired from a first imaging device, and the first captured image is transmitted to a second display device. In addition, an image communication apparatus is provided that includes a control unit that transmits user information of the second display device to the first display device on the first imaging device side.

  The user information of the second display device is a second captured image including a face image of the user of the second display device acquired by the second imaging device on the second display device side. May be.

  The image communication apparatus further includes an image processing unit that performs image processing to reduce a difference in detail between the face image area included in the first captured image and the face image included in the second captured image, The control unit transmits the first captured image after the image processing by the image processing unit to the second display device, and the second captured image after the image processing is transmitted to the first display device. You may send it to.

  The first display device is a display device associated with person identification information acquired from a face image included in the first captured image in a database in which person identification information and a display device are associated with each other. Also good.

  When the detail level of the face image included in the first captured image transmitted to the second display device exceeds a threshold, the control unit associates with the person specifying information acquired from the face image. The user information of the second display device may be transmitted to the first display device.

  When the face image of the user of the second display device is not included in the second captured image, the control unit directs user information of the second display device to the first display device. You do not have to send it.

  The image processing may be processing for aligning the details of the other face image regions with the details of the face image region having the lowest detail among the face image regions of the plurality of captured images.

  The image processing unit may further align the frame rate of another captured image to the frame rate of the captured image having the lowest frame rate among the plurality of captured images.

  When the first captured image includes a plurality of face images, the image processing unit is configured to reduce the difference between each of the plurality of face images and the detail level of the second face image. The captured image is subjected to image processing for each of the plurality of face images, and the control unit obtains each of the second captured image that has been subjected to image processing for each of the plurality of face images from each of the plurality of face image regions. The information may be transmitted to the first display device associated in the database with the person specifying information.

  The position where the first display device is visible is included in the imaging range of the first imaging device, and the position where the second display device is visible is included in the imaging range of the second imaging device. Also good.

  In order to solve the above-described problem, according to another aspect of the present invention, a step of acquiring a first captured image from a first imaging device, and the first captured image on a second display device There is provided an image communication method including a step of transmitting toward the first display device on the first imaging device side, and a step of transmitting user information of the second display device toward the first display device on the first imaging device side.

  In order to solve the above problem, according to another aspect of the present invention, a computer acquires a first captured image from a first imaging device, and the first captured image is displayed as a second display device. In the case of transmitting to the user, a program for causing the user information of the second display device to function as a control unit that transmits the user information to the first display device on the first imaging device side is provided.

  In order to solve the above-described problem, according to another aspect of the present invention, the first imaging device, the first display device provided on the first imaging device side, the first imaging device to the first imaging device. An image communication system including an image communication device that acquires one captured image and a second display device, wherein the image communication device transmits the first captured image to a second display device There is provided an image communication system having a control unit that transmits user information of the second display device to the first display device on the first imaging device side.

  As described above, according to the present invention, it is possible to support remote communication while protecting privacy.

It is explanatory drawing which showed the structure of the image communication system by embodiment of this invention. It is explanatory drawing which showed the detailed structure of the image communication system. It is the top view which showed the relationship between the viewing angle of the display part of a viewer side apparatus, and the angle of view of an imaging part. It is explanatory drawing which showed the specific example of the display screen of a to-be-viewed person side display apparatus. It is explanatory drawing which showed the specific example of the image adjustment process. It is a sequence diagram which shows operation | movement of the image communication system by this embodiment. It is the flowchart which showed the operation | movement regarding the image adjustment process by an image processing communication apparatus. It is explanatory drawing which showed the specific example of the captured image of the base B. FIG. It is explanatory drawing which showed the modification of the display screen of a to-be-viewed person side display apparatus.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  In the present specification and drawings, a plurality of components having substantially the same functional configuration may be distinguished by adding different alphabets after the same reference numeral. For example, a plurality of configurations having substantially the same functional configuration are distinguished as the viewer-side display devices 200A and 200B as necessary. However, when it is not necessary to particularly distinguish each of a plurality of constituent elements having substantially the same functional configuration, only the same reference numerals are given. For example, when there is no need to distinguish between the browsing-side display devices 200A and 200B, they are simply referred to as the browsing-side display device 200.

<1. Basic Configuration of Image Communication System>
First, the basic configuration of the image communication system according to the present embodiment will be described with reference to FIG.

  FIG. 1 is an explanatory diagram showing a configuration of an image communication system according to an embodiment of the present invention. As shown in FIG. 1, the image communication system according to the embodiment of the present invention includes a viewer-side device 100, a viewer-side display device 200, a viewer-side imaging device 300, and an image processing communication device 400. . In FIG. 1, a base A and a base B are shown as a plurality of remote spaces, the viewer-side device 100 exists at the base A, and the viewer-side display device 200 and the viewer-side imaging at the base B are shown. A device 300 is present.

  The viewer-side imaging device 300 (first imaging device) captures the base B and acquires the captured image of the base B. Then, the captured image of the base B acquired by the viewer-side imaging device 300 is transmitted to the viewer-side device 100 via the image processing communication device 400 in response to a request from the base A, for example. In addition, since the captured image of the base B is an image that can be browsed in a remote space, a person (subject) included in the captured image of the base B is hereinafter referred to as a “viewed person”.

  As shown in FIG. 1, the viewer-side device 100 includes an imaging unit 101, a display unit 103, and an input unit 104. The imaging unit 101 (second imaging device) images the user of the viewer-side device 100. The display unit 103 (second display unit) displays a captured image of the base B received from the viewed person side imaging device 300 via the image processing communication device 400. The input unit 104 receives an operation input for controlling the viewer-side imaging device 300. For example, in the case where the viewer-side imaging device 300 has an optical zoom function, an angle of view change function, and the like, it is possible to specify zoom processing and angle of view change processing by an operation input to the input unit 104. In addition, since the user of the viewer side apparatus 100 can browse the captured image of the base B by the act of looking at the display unit 103 of the viewer side apparatus 100, in the following, the viewer side apparatus 100 present at the base A A user is referred to as a “viewer”.

  When there is a viewer of the captured image of the base B, the image processing communication device 400 transmits, for example, a captured image of the viewer to the viewed user display device 200 as the viewer information (user information). Then, the viewer-side display device 200 (first display device) displays the viewer image received from the image processing communication device 400. As described above, according to the present embodiment, the viewer information is notified to the viewer side, so that the viewer can determine whether he / she is currently watching or who is watching. . As a result, it is possible to suppress a secret unilateral remote browsing action that leads to a privacy problem. Furthermore, the image processing communication apparatus 400 according to the present embodiment suppresses the above problem by adjusting the information amount of the viewer image or the captured image of the base B as necessary, and the viewer and the viewer are the same. It is possible to realize a natural sense of communication as if it exists in a space. This point will be described in detail later.

  In addition, in FIG. 1, a usage form in which one viewer owns one viewer-side device 100 and one viewer owns one viewer-side display device 200 is shown. However, the present embodiment can also be applied to a usage pattern in which a single device is shared by a plurality of users.

  In addition, FIG. 1 shows an example in which there is one viewer imaged by the viewer-side device 100, but a viewer imaged by the viewer-side device 100, that is, a person existing at the base A May be plural.

  Similarly, in FIG. 1, an example is shown in which one viewer is captured by the viewer-side imaging device 300, but the viewer who is captured by the viewer-side imaging device 300, that is, A plurality of persons may exist at the base B. In particular, since the viewer-side imaging device 300 captures a bird's-eye view image of the base B, there is a high possibility that a plurality of subjects (viewees) exist within the angle of view. It is possible to notify the viewer information to the viewer-side display device 200 that each viewer individually has.

  As described above, the present invention can also be applied to a usage mode in which there are a plurality of viewers at the base A and a viewer at the base B, but in the following, the simplest one viewer is used. Yes, the explanation will be given with emphasis on the usage form in which there is only one person to be viewed.

<2. Detailed configuration of image communication system>
The basic configuration of the image communication system according to the present embodiment has been described above with reference to FIG. Subsequently, the viewer-side device 100, the viewer-side display device 200, the viewer-side imaging device 300, and the image processing communication device 400 that constitute the image communication system will be sequentially described in detail.

[2-1. Viewer side device]
FIG. 2 is an explanatory diagram showing a detailed configuration of the image communication system. As shown in FIG. 2, the viewer-side device 100 configuring the image communication system includes an imaging unit 101, a face detection unit 102, a display unit 103, an input unit 104, and a control unit 105. 2 shows an example in which each configuration of the imaging unit 101, the face detection unit 102, the display unit 103, the input unit 104, and the control unit 105 is mounted on one viewer-side device 100. The configuration may be distributed in a plurality of devices.

  The imaging unit 101 captures a viewer included in the imaging range, acquires a browser image, and outputs the browser image to the face detection unit 102. Note that the imaging unit 101 may be a general imaging device including an imaging optical system and an imaging element, such as a digital camera or a Web camera.

  The face detection unit 102 detects the viewer's face area from the viewer image acquired by the imaging unit 101, and outputs the face area information indicating the viewer's face area and the viewer image to the control unit 105. This face area information is, for example, data in which a face area in a viewer image and an area other than a face are expressed in binary, and indicates the position, size, and shape of the face area in the viewer image. As an image processing method for detecting a face area, there are a plurality of existing methods such as a method using skin color information and a method of paying attention to facial parts. In this embodiment, any of these methods is used. May be used. Note that since the viewer device 100 does not require a person identification, that is, a collation function between the viewer and the face image database, the face detection unit 102 does not have a function of recognizing the features of the viewer's face. Good.

  The display unit 103 displays a captured image of the base B acquired by the imaging unit 301 of the viewed person side imaging device 300. Note that the captured image of the site B is input via the control unit 401 of the image processing communication device 400 and the control unit 105 of the viewer-side device 100. The display unit 103 may be a general display device such as a liquid crystal display or an organic EL display. However, it is desirable that the display unit 103 is arranged at a position where a position where the display unit 103 can be visually recognized is included in the imaging range of the imaging unit 101. Hereinafter, this point will be specifically described with reference to FIG.

  FIG. 3 is a plan view showing the relationship between the viewing angle of the display unit 103 and the angle of view of the imaging unit 101. In FIG. 3, the viewing angle of the display unit 103 (a range in which the display unit 103 can be viewed) is indicated by a broken line, and the angle of view (imaging range) of the imaging unit 101 is indicated by a one-dot chain line. In order to realize the effect of the present embodiment, it is important to accurately detect that the browsing action of the viewer has occurred. For this reason, as shown in FIG. 3, when the viewer browses the display unit 103, it is desirable that the viewer's face should basically fall within the angle of view of the imaging unit 101.

  Furthermore, if the display unit 103 and the imaging unit 101 are arranged so that the viewing angle of the display unit 103 is within the angle of view of the imaging unit 101, when the viewer views the display unit 103, that is, the captured image of the base B, A viewer image including the viewer's face is always obtained, and a face region can be detected from the viewer image. Such an arrangement relationship can also be realized by using a general liquid crystal display with a built-in web camera in which a web camera is embedded in the upper frame of the liquid crystal display. Similarly, the imaging unit 301 of the viewer-side device 300 and the display unit 201 of the viewer-side device 200 are similarly arranged at positions where the position where the display unit 201 can be viewed is included in the imaging range of the imaging unit 301. Desirable to be

  The input unit 104 receives an operation input for controlling the viewer-side imaging device 300. For example, in the case where the viewer-side imaging device 300 has an optical zoom function, an angle of view change function, and the like, it is possible to specify zoom processing and angle of view change processing by an operation input to the input unit 104. In addition, the viewer can request transmission of a captured image of the base B by an operation input to the input unit 104. Note that if the viewer-side imaging apparatus 300 does not have an optical zoom function, a view angle changing function, or the like, or if the viewer does not have a desire for an operation such as zooming, the zoom process or the view angle changing process is performed. The input unit 104 for designating is not necessary.

  The control unit 105 controls the overall operation of the viewer-side device 100. For example, the control unit 105 transmits a viewer image and face area information input from the face detection unit 102 to the image processing communication device 400. Further, the control unit 105 outputs the captured image of the base B received from the image processing communication device 400 to the display unit 103. In addition, the control unit 105 transmits instruction information input from the input unit 104 to the image processing communication device 400.

[2-2. Reader-side display device]
As shown in FIG. 2, the viewer-side display device 200 includes a display unit 201 and a control unit 202. The viewer-side display device 200 may be an information terminal such as a smartphone, a mobile phone, a portable game machine, or a portable video processing device.

  The display unit 201 displays a viewer image acquired by the viewer-side device 100. The viewer image is input via the control unit 401 of the image processing communication apparatus 400 and the control unit 202 of the viewed person display device 200. Hereinafter, the viewer image displayed on the display unit 201 will be described more specifically with reference to FIG.

  FIG. 4 is an explanatory diagram showing a specific example of a viewer image displayed on the display unit 201. As shown in FIG. 4, the display unit 201 includes a content display area 210 and a viewer image display area 220.

  In the content display area 210, content for a normal use of the viewed-side display device 200 is displayed. Although FIG. 4 shows a calendar as an example of content, the content may be a photo, a video, a game, or various operation screens.

  In addition, a viewer image 222 is displayed in the viewer image display area 220. As will be described in detail later, this viewer image has the same amount of information as the face area of the viewer included in the captured image of the base B displayed on the display unit 103 of the viewer-side device 100.

  With such a configuration, when the viewer-side display device 200 is not a terminal only for the use of the present embodiment but an information terminal that is used for general purposes, the content display area 210 has a purpose for the original use. The viewer image can be displayed in the viewer image display area 220 while displaying the content.

The display unit 201 may be a general display device such as a liquid crystal display or an organic EL display, like the display unit 103 of the viewer-side device 100.

  The control unit 202 controls the overall operation of the viewer-side display device 200. For example, the control unit 202 outputs a viewer image received from the image processing communication device 400 to the display unit 201.

[2-3. Reader-side imaging device]
As shown in FIG. 2, the viewer-side imaging device 300 includes an imaging unit 301, a face detection unit 302, a face recognition unit 303, and a control unit 304. In FIG. 2, the viewer-side imaging device 300 and the viewer-side display device 200 are described as separate devices, but the functions of the viewer-side imaging device 300 and the viewer-side display device 200 are described. The function may be implemented in the same device. Further, FIG. 2 illustrates an example in which each configuration of the imaging unit 301, the face detection unit 302, the face recognition unit 303, and the control unit 304 is mounted on one viewer-side imaging device 300. The configuration may be distributed in a plurality of devices.

  The imaging unit 301 captures a subject included in the imaging range, acquires a captured image of the base B, and outputs the captured image of the base B to the face detection unit 302. Note that the imaging unit 301 may be a general imaging device including an imaging optical system and an imaging element, such as a digital camera or a Web camera.

  The face detection unit 302 detects the face area of the person being viewed from the captured image of the base B acquired by the image pickup part 301, and acquires the feature information of the face image of the person being viewed. Then, the face detection unit 302 outputs face area information indicating the face area of the person to be viewed, face image feature information, and a captured image of the base B to the face recognition unit 303.

  The feature information of the face image is feature amount / pattern data based on the relative position, size, and shape of each part of the face. As the feature quantity, there are various existing feature quantities such as the Haar wavelet feature quantity. In the present embodiment, any of these feature quantities may be used.

  The face recognition unit 303 identifies the person to be viewed from the feature information of the face image obtained by the face detection unit 302, and outputs the captured image of the base B, face area information, and person identification information to the control unit 304. . Note that the facial image feature information for specifying a person has been registered in advance, that is, a facial image database for collation may be prepared in advance. In addition, algorithms used for face recognition and person identification and data processing methods are not particularly limited.

  The control unit 304 controls the overall operation of the viewer-side imaging device 300. For example, the control unit 304 transmits the captured image of the base B, the face area information, and the person specifying information input from the face recognition unit 303 to the image processing communication device 400. In addition, the control unit 304 receives operation instruction information from the input unit 104 of the viewer-side device 100 via the control unit 401 of the image processing communication device 400, and executes zoom processing and angle of view change processing according to the instruction. To do.

[2-4. Image processing communication device]
As illustrated in FIG. 2, the image processing communication apparatus 400 includes a control unit 401, a storage unit 402, and an image processing unit 403.

  The control unit 401 controls the overall operation of the image processing communication apparatus 400. For example, the control unit 401 receives a viewer image and face area information at the site A from the viewer-side device 100 and outputs the received image to the storage unit 402. In addition, the control unit 401 receives the captured image of the base B, the face area information, and the person specifying information from the viewer-side imaging device 300, and outputs them to the storage unit 402.

  Further, the control unit 401 compares the face area information of the viewer image received from the viewer-side apparatus 100 with the face area information of the viewed person received from the viewed-side image pickup apparatus 300, and performs image processing. Via the storage unit 402, the unit 403 is instructed to perform image adjustment processing for reducing the difference between the information amount of the face area of the viewer image and the information amount of the face area of the viewer. Details of this image adjustment processing will be described later with reference to FIG.

  In addition, the control unit 401 transmits the viewer image after the image adjustment process input from the storage unit 402 to the viewed-person-side display device 200 that is personally owned by the viewed person based on the person specifying information of the viewed person. To do. In addition, the control unit 401 transmits the captured image of the base B input from the storage unit 402 to the viewer-side device 100.

  The storage unit 402 stores the browser image, the viewer's face area information, the captured image of the base B, the browse person's face area information, and the browse person's person specifying information input from the control unit 401. Further, the storage unit 402 outputs the image adjustment processing instruction and the viewer image input from the control unit 401 to the image processing unit 403. Further, the storage unit 402 outputs at least one of the viewer image after the image adjustment processing output from the image processing unit 403 or the captured image of the base B to the control unit 401.

  The image processing unit 403 performs image adjustment processing to reduce the difference between the information amount of the face area of the viewer image and the information amount of the face area of the person to be browsed according to the image adjustment processing instruction input from the storage unit 402. . The amount of information in this specification indicates not the amount of data in the image file but the level of detail of information for a person viewing an image, and the amount of information in this specification corresponds to, for example, the number of pixels. Here, the image adjustment processing when the information amount is the number of pixels will be described in more detail.

(Significance of image adjustment processing)
In the case of direct visual observation in the same space, the first person and the second person recognize each other's appearance at substantially the same scale. That is, the information amount of the second person obtained by the first person is equal to the information amount of the first person obtained by the second person.

  However, in communication between remote spaces, the amount of information of the second person obtained by the first person is expected to be different from the amount of information of the first person obtained by the second person. For example, when the number of pixels of the face area of the viewer image at the base A is 200 and the number of pixels of the face area of the viewer included in the captured image at the base B is 100, the face area of the viewer is viewed The amount of information is about twice that of the person's face area. At this time, if the image processing communication device 400 transmits the viewer image as it is to the viewer-side display device 200, the information amount of the viewer's face area obtained by the viewer is obtained by the viewer. This is twice the amount of information in the viewer's face area.

  Therefore, in this embodiment, in order to reproduce this natural feeling even in communication between remote spaces, an image that reduces the difference between the information amount of the face area of the viewer image and the information amount of the face area of the viewer Perform the adjustment process.

(Target of image adjustment processing)
Specifically, the image processing unit 403 processes at least one of the viewer image and the captured image of the base B, thereby reducing the difference in the information amount of the face area included in both. For example, the image processing unit 403 performs image processing for aligning the information amount of the image having the larger face region information amount with the other information amount among the viewer image and the captured image of the base B.

  Here, in the assumed usage mode, the imaging unit 101 is arranged in the vicinity of the display unit 103, and accordingly, the viewer is positioned near the imaging unit 101, and as a result, the viewer's face area may be enlarged. high. On the other hand, since the imaging unit 301 at the site B captures a bird's-eye view image, the viewer is positioned away from the imaging unit 301, and as a result, the face area of the viewer may be smaller than the viewer. Is expensive.

  For this reason, in many cases, it is considered that the image processing unit 403 performs image processing that aligns the information amount of the viewer's face area with the information amount of the person's face area. However, when the information amount of the viewer's face area included in the browser image is smaller than the information amount of the person's face area included in the captured image of the base B, the image processing unit 403 applies the captured image of the base B to the captured image. You may perform the image process which aligns the information content of the to-be-viewed person's face area with the information content of a viewer's face area. In this case, the image processing unit 403 may reduce only the information amount of the face area of the person to be viewed in the captured image of the base B, or may reduce the information amount of the entire captured image of the base B.

  As a modified example, the image processing unit 403 processes both the viewer image and the captured image of the base B regardless of the information amount of the viewer's face area and the information amount of the person's face area. The information amount of the person's face area and the information amount of the person's face area may be aligned with the same information amount.

(Image adjustment processing method)
In addition, the image processing unit 403 converts the resolution of the viewer image or the captured image of the base B using an existing method such as the nearest neighbor method or the bicubic method, so that the information amount of the face area of both images Align.

  For example, when the number of pixels of the viewer's face area is four times the number of pixels of the person's face area included in the captured image of the base B, the image processing unit 403, as shown in FIG. A process of reducing the number of pixels of the viewer image to a quarter is performed. This is equivalent to reducing the information amount of the image to a quarter.

  It should be noted that the information amounts of both face areas do not need to be completely matched, and the information amounts of both face areas after the image adjustment processing may have a difference within a predetermined range. In the above description, resolution conversion processing has been described as an information amount conversion method. However, as an application example, it is possible to reduce the information amount by blurring an image as in blurring processing. Further, the image processing unit 403 may convert the other frame rate to the frame rate of the image with the lower frame rate among the viewer image and the captured image of the site B from the viewpoint of fairness of the information amount. Good.

<3. Operation of Image Communication System>
The configuration of the image communication system according to the present embodiment has been described above. Subsequently, the operation of the image communication system according to the present embodiment will be described.

(Overall flow)
FIG. 6 is a sequence diagram showing the operation of the image communication system according to the present embodiment. As illustrated in FIG. 6, the imaging unit 301 of the viewed person side imaging apparatus 300 starts imaging of the base B (S110). In addition, since the imaging part 301 plays the role of the live camera which images the bird's-eye view scenery of the base B, the usage form which always images is basically assumed.

  Subsequently, the face detection unit 302 of the viewer-side imaging device 300 detects the viewer's face area from the captured image of the base B acquired by the imaging unit 301, and obtains feature information of the viewer's face image. get. Then, the face detection unit 302 outputs the face area information indicating the face area of the person to be viewed, the feature information of the face image, and the captured image of the base B to the face recognition unit 303 (S112).

  Here, when the face area is not detected, that is, when there is no person to be viewed as a subject within the angle of view of the imaging unit 301, or when there is a person to be viewed but no face is reflected, the face detection unit 302 may output information that the face area is not detected instead of the area information. Supplementing this point, it can be considered that the subject part to be most considered in the privacy problem to be solved by the present embodiment is the face. For this reason, if it is considered that the person to be viewed exists but the face is not reflected, for example, if the person to be viewed is moving backward with respect to the imaging unit 301, treat it as out of the problem of unilateral browsing. Is appropriate. This is because even if the browsing action in the same space that is not remote is viewed from behind, the person to be browsed is not aware, so the nature of the problem is different from the problem of the unilateral browsing action that is peculiar to communication in the remote space.

  Then, the face recognition unit 303 of the image capturing apparatus 300 on the viewed person side identifies the person to be viewed from the feature information of the face image obtained by the face detecting unit 302, and the captured image of the base B, the face area information, The person specifying information is output to the control unit 304 (S114). Next, the control unit 304 of the viewed person side imaging apparatus 300 starts transmission of the captured image of the base B, face area information, and person specifying information to the image processing communication apparatus 400 (S116).

  On the other hand, the imaging unit 101 of the viewer-side device 100 starts imaging a viewer included in the imaging range and acquires a viewer image (S118). Then, the face detection unit 102 of the browser-side device 100 detects the viewer's face area from the browser image acquired by the imaging unit 101, and controls the face area information and the viewer image indicating the viewer's face area. The data is output to the unit 105 (S120).

  Here, when the face area is not detected, that is, when there is no viewer of the display unit 103 or when the viewer does not look at the screen direction, the face detecting unit 102 has not detected the face area. Information to that effect may be output instead of the area information. Even if the face detection unit 102 does not explicitly output information indicating that no face area has been detected as described above, the control unit 105 does not need to output only information indicating a non-face image range. This makes it possible to recognize that no face area has been detected.

  Thereafter, the control unit 105 of the viewer-side device 100 starts transmitting the browser image and face area information to the image processing communication device 400 together with the request for the captured image of the base B (S122).

  The image processing communication device 400 starts image adjustment processing based on the various data received from the viewer-side device 100 and the various data received from the viewer-side imaging device 300 (S130). The operation flow of this image adjustment process will be described later with reference to FIG.

  Thereafter, the image processing communication device 400 starts transmitting a viewer image to the viewed-subject display device 200 (S146), and starts transmitting a captured image of the base B to the viewer-side device 100 (S148). . As a result, the display unit 201 of the viewer-side display device 200 displays the viewer image (S150), and the display unit 103 of the viewer-side device 100 can display the captured image of the base B (S152). ). In addition, in order to prevent a viewer's one-side browsing action, it is desirable that the transmission start timing of the captured image of the base B and the transmission start timing of the viewer image are synchronized.

(Operation flow of image adjustment processing)
FIG. 7 is a flowchart showing operations related to image adjustment processing by the image processing communication apparatus 400. As shown in FIG. 7, first, the control unit 401 of the image processing communication device 400 receives the viewer image and face area information of the base A from the viewer-side device 100 and receives the base image from the viewer-side imaging device 300. The captured image of B, face area information, and person specifying information are received (S131).

  Subsequently, the control unit 401 determines whether or not a face area has been detected from the viewer image based on the viewer's face area information (S132). If it is determined that a face area has been detected, the process proceeds to S133. If it is determined that no face area has been detected, the process proceeds to S138.

  Similarly, the control unit 401 determines whether or not a face area is detected from the captured image of the base B based on the face area information of the person to be viewed (S133). If it is determined that a face area has been detected, the process proceeds to step S134. If it is determined that no face area has been detected, the process proceeds to step S138.

  When it is determined that the face area is detected from the viewer image and the captured image of the base B, the control unit 401 compares the face area of the viewer with the face area of the person to be browsed, and the image processing unit 403 Via the storage unit 402, an instruction is given for an image adjustment process for reducing the difference between the information amount of the face area of the viewer image and the information amount of the face area of the viewer (S134).

  Subsequently, the storage unit 402 stores the viewer image of the base A, the face area information of the viewer, the captured image of the base B, the face area information of the reader, and the person specifying information. These data are held until the transmission of the captured image of the site B is completed (S135).

  Then, the image processing unit 403 performs image adjustment processing to reduce the difference between the information amount of the face area of the viewer image and the information amount of the face region of the viewed person in accordance with the image adjustment processing instruction from the control unit 401. (S136).

  Thereafter, the control unit 401 transmits the viewer image after the image adjustment processing to the viewer-side display device 200 (S146). Here, each viewer at the base B has a specific viewer-side display device 200, and the viewer and the viewer-side display device 200 are associated one-to-one in advance in the database. It shall be. The association method is not particularly limited. In addition, one viewer may own a plurality of viewer-side display devices 200, and a plurality of viewer-side display devices 200 may be associated with one viewer. The control unit 401 uses this database to identify the viewed person side display device 200 corresponding to the person identification information received from the viewed person side imaging device 300, and performs image adjustment on the identified viewed person side display device 200. Send the processed viewer image.

  In addition, the control unit 401 transmits the captured image of the site B to the viewer-side device 100 (S148). Thereby, when a browsing person browses the captured image of the base B, it becomes possible to display a browsing person image on the display part 201 of the to-be-viewed person side display apparatus 200. FIG. In addition, in order to prevent a viewer's one-side browsing action, it is desirable that the transmission start timing of the captured image of the base B and the transmission start timing of the viewer image are synchronized.

<4. Modification>
(Modification 1)
In addition, according to said flow, even when it is thought that there is no browsing act of the captured image of the base B, when the face area is not detected from a viewer image, the captured image of the base B is a viewer side apparatus. 100. This is a system in which the image communication system according to the present embodiment supports informal remote communication, and basically the purpose of the system is to always continue to flow images of remote bases regardless of whether or not there is a browsing act. It is a specification based on the idea that it is effective against. However, when emphasizing the viewpoint of saving power consumption or reducing the amount of communication data, image processing is performed so that the captured image of the base B is transmitted to the viewer-side apparatus 100 only when the viewer's face area is detected. The communication device 400 may be designed.

(Modification 2)
Further, when there are a plurality of viewers, the control unit 401 may transmit the viewer images after the image adjustment processing of each viewer to the viewer-side display device 200. In this case, the viewed person display device 200 displays each viewer image on the display unit 201 so as not to overlap.

  Further, when a plurality of viewers are included in the captured image of the base B, the control unit 401 may transmit the viewer images after the image adjustment processing to the viewer-side display device 200 of each viewer. . Hereinafter, this point will be described more specifically with reference to FIG.

  FIG. 8 is an explanatory diagram showing a specific example of a captured image of the base B. The captured image of the base B shown in FIG. 8 includes a person A having a viewed person display device 200A and a person B having a viewed person display device 200B. Here, it is assumed that the number of pixels of the face area of the person A is 200 and the number of pixels of the face area of the person B is 50. In this case, the image processing unit of the image processing communication device 400 creates a viewer image whose face area has 200 pixels for transmission to the viewer-side display device 200A, and also displays the viewer-side display. For transmission to the apparatus 200B, a viewer image in which the number of pixels of the face area is 50 is created. Then, the control unit 401 transmits a viewer image whose face area has 200 pixels to the person A to-be-reader display device 200A, and transmits a viewer image whose face area has 50 pixels to the person B. It transmits to the to-be-viewed person side display apparatus 200B.

  Thus, this embodiment operates without trouble even when there are a plurality of viewers at the base A and at a base B. This can be said to satisfy important operating conditions in a remote communication support system, particularly in a system assuming an office-to-office telework environment in which a plurality of members are distributed at a plurality of remote bases.

(Modification 3)
In FIG. 4, the viewer image has been described as an example of the viewer information notified to the viewer-side display device 200 of the viewer, but other information is used as the viewer information. May be notified. For example, when the viewer-side device 100 or the image processing communication device 400 has the function of specifying the viewer's person, the image processing communication device 400 notifies the viewer-side display device 200 of the name of the viewer as the viewer information. May be. Further, the image processing communication device 400 may notify the viewer-side display device 200 together with information indicating the level of detail of the viewer's face area included in the captured image of the base B.

  With this configuration, the display unit 201 of the viewer-side display device 200 replaces the viewer image with the viewer's name 222 and the viewer's attention level (detail level of the face area) as shown in FIG. Can be displayed.

(Modification 4)
Moreover, although the above demonstrated the example which notifies browser information to the browsing person side display apparatus 200 of a browsing person when the browsing person is browsing the captured image of the base B containing a browsing person, Embodiments are not limited to such examples. For example, even if a person image is included in the captured image of the base B, when the information amount of the person image is small (when the person image is small), the viewer does not intentionally view the person image. That is, the person is regarded as not being viewed. Therefore, when the information amount of the person image (or face area) included in the captured image of the base B is less than the threshold, the image processing communication apparatus 400 displays the viewer information on the person-to-be-viewed person side display device 200. If the amount of information exceeds a threshold value without notification, the viewer information may be notified to the person-to-be-viewed person side display device 200.

<5. Conclusion / Supplement>
As described above, according to the present embodiment, the “browser information” indicating “who” is viewing the person being viewed with “what amount of information” is notified to the person being viewed. As a result, in the environment where each site space video is always transmitted and received mutually for the purpose of sharing informal situation information in a remote distributed collaborative environment, a privacy problem that occurs inevitably, that is, a secret one Remote browsing behavior can be suppressed. This embodiment is also useful in that it enables the construction of a “fair and intuitive relationship of seeing and seeing” similar to the case where the viewer and the viewer are in the same space even in a remote situation. .

  Although the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

  For example, in the above description, an example is described in which the viewer-side device 100 is disposed at the base A, and the viewer-side display device 200 and the viewer-side imaging device 300 are disposed at the base B. The viewer at the base A and the viewer at the base B may have a portable image processing terminal in which the function of the face recognition unit 303 is added to the viewer-side device 100. Thereby, it becomes possible to apply this embodiment also to the utilization form in which a viewer and a to-be-read person move freely. In addition, since the viewer and the person to be browsed have the same image processing terminal, the viewer at the base A may be the viewer, and the viewer at the base B may be the viewer. It becomes possible easily and the correspondence to the bidirectionality of the exchange of information is realized. This indicates the “scalability” of the system of the present embodiment, that is, the high scalability, together with the ease of handling when there are a plurality of viewers and viewers.

  Here, as the above-described image processing terminal, a smart phone that is often provided with an imaging unit, a display unit, a face detection unit, and a face recognition unit in a single device is suitable. However, in this case, since it is desired that the image processing terminal of the viewed person unplannedly captures the viewed person within the angle of view, in order to stably provide the situation information of the space, It is desirable in terms of operation that the browsing-side imaging device 300 that always captures images from above is present at both sites together with the image processing terminal.

  In addition, each step in the processing of the image communication system does not necessarily have to be processed in time series in the order described as a sequence diagram or a flowchart. For example, each step in the processing of the image communication system may be processed in an order different from the order described as the flowchart, or may be processed in parallel.

  In addition, hardware such as a CPU, a ROM, and a RAM built in the viewer-side device 100, the viewer-side display device 200, the viewer-side imaging device 300, and the image processing communication device 400 are the above-described viewer side. It is also possible to create a computer program for exhibiting functions equivalent to the configurations of the device 100, the viewer-side display device 200, the viewer-side imaging device 300, and the image processing communication device 400. A storage medium storing the computer program is also provided.

DESCRIPTION OF SYMBOLS 100 Browser side apparatus 101 Imaging part 102 Face detection part 103 Display part 104 Input part 105 Control part 200 Browse person side display apparatus 201 Display part 202 Control part 300 Browse person side imaging apparatus 301 Imaging part 302 Face detection part 303 Face Recognition unit 304 Control unit 400 Image processing communication device 401 Control unit 402 Storage unit 403 Image processing unit

Claims (10)

When the first captured image is acquired from the first imaging device and the first captured image is transmitted to the second display device, the user information of the second display device is transmitted to the first display device. A control unit that transmits the image to the first display device on the imaging device side;
With
The user information of the second display device is a second captured image including a face image of the user of the second display device acquired by the second imaging device on the second display device side. ,
The degree of detail of the face image area included in the first captured image and the face image area included in the second captured image is set to a plurality of captured images including the first captured image and the second captured image. An image processing unit that performs image processing to match the detail level of the face image region with the lowest level of detail among the face image regions of
The control unit transmits the first captured image after the image processing by the image processing unit to the second display device, and the second captured image after the image processing is transmitted to the first display device. Send to
Image communication device. The first display device is a display device associated with person identification information acquired from a face image included in the first captured image in a database in which person identification information and a display device are associated with each other. The image communication apparatus according to claim 1 . When the number of pixels of the face image included in the first captured image transmitted to the second display device exceeds a threshold value, the control unit associates with the person specifying information acquired from the face image. It is to send the user information of the second display device toward the first display device and an image communication apparatus according to claim 1 or 2. When the face image of the user of the second display device is not included in the second captured image, the control unit directs user information of the second display device to the first display device. The image communication apparatus according to claim 1 , wherein the image communication apparatus is not transmitted. The image communication device according to claim 1 , wherein the image processing unit further aligns the frame rate of another captured image with the frame rate of the captured image having the lowest frame rate among the plurality of captured images. When the first captured image includes a plurality of face image areas, the image processing unit aligns the details of each of the plurality of face image areas and the face image area of the second captured image. For each of the plurality of face image regions,
The control unit associates each of the second captured images subjected to image processing for each of the plurality of face images with person specifying information acquired from each of the plurality of face image regions in the database. The image communication device according to claim 2 , wherein the image communication device transmits the image to one display device. The position where the first display device can be visually recognized is included in the imaging range of the first imaging device,
It said second position visible display device is included in the imaging range of the second imaging apparatus, the image communication apparatus according to any one of claims 1-6. Obtaining a first captured image from the first imaging device;
Obtaining a second captured image from the second imaging device;
The degree of detail of the face image area included in the first captured image and the face image area included in the second captured image is set to a plurality of captured images including the first captured image and the second captured image. Performing image processing to match the detail level of the face image area with the lowest level of detail among the face image areas of
Transmitting the first captured image after the image processing toward a second display device, and transmitting the second captured image after the image processing toward the first display device;
Including an image communication method.
Image communication method. Computer
When the first captured image is acquired from the first imaging device and the first captured image is transmitted to the second display device, the user information of the second display device is transmitted to the first display device. A control unit that transmits the image to the first display device on the imaging device side;
With
The user information of the second display device is a second captured image including a face image of the user of the second display device acquired by the second imaging device on the second display device side. ,
The degree of detail of the face image area included in the first captured image and the face image area included in the second captured image is set to a plurality of captured images including the first captured image and the second captured image. An image processing unit that performs image processing to match the detail level of the face image region with the lowest level of detail among the face image regions of
The control unit transmits the first captured image after the image processing by the image processing unit to the second display device, and the second captured image after the image processing is transmitted to the first display device. Send to
A program for functioning as an image communication apparatus . An image including a first imaging device, a first display device provided on the first imaging device side, an image communication device that acquires a first captured image from the first imaging device, and a second display device A communication system,
The image communication device includes:
When transmitting the first captured image toward the second display device, the control unit transmits user information of the second display device toward the first display device on the first imaging device side. Have
The user information of the second display device is a second captured image including a face image of the user of the second display device acquired by the second imaging device on the second display device side. ,
The image communication device includes:
The degree of detail of the face image area included in the first captured image and the face image area included in the second captured image is set to a plurality of captured images including the first captured image and the second captured image. An image processing unit that performs image processing to match the detail level of the face image region with the lowest level of detail among the face image regions of
The control unit transmits the first captured image after the image processing by the image processing unit to the second display device, and the second captured image after the image processing is transmitted to the first display device. An image communication system that transmits to

Images