JP2018073093A - Display controlling system, image processing device, display controlling method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display controlling system, an image processing device, a display controlling method, and a program capable of avoiding unnatural display of a pointing image.SOLUTION: A remote communication system includes a first image processing device 101a and a second image processing device 101b. Display controlling means is further provided for controlling the second image processing device to display a pointing image 105a indicating an action of a user C pointing out an image 105b on a display region 103a of the first image processing device. In a projection region 103b of the second image processing device 101b, a pointing image 106b is projected from a side of the second image processing device 101b where no user C is present.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a display control system, an image processing device, a display control method, and a program, and more particularly to a display control system, an image processing device, a display control method, and a program that perform remote communication.

  A display control system for displaying each image is known as means for communicating with a remote place (remote place) (hereinafter referred to as “remote communication”). In the display control system, for example, an image processing apparatus including a display unit and an audio output unit is arranged in each of two separated conference rooms A and B, and the image processing apparatuses communicate with each other (hereinafter referred to as “remote communication communication”). )I do. In each conference room A, B, the voice of the communication destination is output from the audio output unit of each image processing device, and the images of the users in each conference room A, B, shared electronic materials, etc. Is displayed (for example, see Patent Document 1). In the display control system, when the shared electronic materials are displayed in the same way in the conference rooms A and B, for example, when the user of the conference room A points to the electronic materials, the conference room B is superimposed on the electronic materials. A pointing image indicating the action indicated by the user in the conference room A is displayed. Thereby, it is possible to easily notify the user of the conference room B of the action that the user of the conference room A points to the electronic material.

  In the display control system, an image processing apparatus including a projector may be used in addition to the display unit. In this case, a user image, electronic material, or the like is projected by the projector. In general, a projector projects each image on a plane perpendicular to the ground, such as a screen or a whiteboard. However, in recent years, the projector is level with respect to the ground such as a desk on which an image processing apparatus is disposed. The number of cases where each image is projected on a plane (hereinafter simply referred to as “horizontal plane”) is also increasing. In this case, the projector displays the electronic material in an easy-to-read direction for the user at the position facing the image processing apparatus. In each of the conference rooms A and B, each projector is located at the same distance from each image processing apparatus, and The electronic material is projected in the same direction with respect to the image processing apparatus. Therefore, in each of the conference rooms A and B, there is a tendency that the electronic material is pointed to by the user at the same position with respect to each image processing apparatus.

JP 2006-41884 A

  However, when the electronic material is pointed to by the user at the same position with respect to each image processing apparatus, for example, in the conference room B, the pointing image of the user in the conference room A is projected on the user's position in the conference room B. Is done. As a result, there arises a problem that an unnatural display appears as if a hand jumps out from the user in the conference room B.

  An object of the present invention is to provide a display control system, an image processing apparatus, a display control method, and a program that can avoid an unnatural display of a pointing image.

  In order to achieve the above object, a display control system of the present invention includes an image processing device and another image processing device, and displays a shared image in each display area of the image processing device and the other image processing device. A display control system for controlling the display of a pointing image indicating an action in which the user of the other image processing device points to the displayed image in a display area of the image processing device; The pointing image has directionality, and the display control means displays the pointing image from the side where the user of the image processing apparatus does not exist in the display area of the image processing apparatus.

  According to the present invention, an unnatural display of a pointing image can be avoided.

It is a figure for demonstrating the remote communication system as a display control system which concerns on embodiment of this invention. FIG. 2 is a perspective view schematically showing a configuration of the projector in FIG. 1. It is a figure for demonstrating the acquisition range of the distance information of the depth sensor of FIG. It is a block diagram which shows the outline of the hardware constitutions of the image processing apparatus of FIG. It is a figure which shows an example of the image projected in the remote communication system of FIG. It is a flowchart which shows the procedure of the remote communication control process performed by the image processing apparatus of FIG. 2 is a flowchart showing a procedure of a pointing image transmission process executed by the image processing apparatus of FIG. 1. It is a flowchart which shows the procedure of the pointing image generation process of step S701 of FIG. It is a figure for demonstrating conversion of the pointing image performed by the image processing apparatus of FIG. 2 is a flowchart illustrating a procedure of a pointing image reception process executed by the image processing apparatus in FIG. 1. It is a flowchart which shows the procedure of the layout information update process of step S1002 of FIG. It is a figure for demonstrating the discrimination | determination in the process of step S1102 of FIG. It is a flowchart which shows the procedure of the 1st modification of the layout information update process of FIG. It is a figure for demonstrating the discrimination | determination in the process of step S1302 of FIG. It is a flowchart which shows the procedure of the 2nd modification of the layout information update process of FIG. It is a figure for demonstrating the discrimination | determination in the process of step S1502 of FIG. It is a flowchart which shows the procedure of the 3rd modification of the layout information update process of FIG. It is a figure for demonstrating the discrimination | determination in the process of step S1702 of FIG. 10 is a flowchart showing respective procedures of a modification of the pointing image reception process of FIG. 10 and a modification of the pointing image transmission process of FIG. 2 is a flowchart showing each procedure of image conversion processing executed by the image processing apparatus of FIG. 1.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram for explaining a remote communication system 100 as a display control system according to an embodiment of the present invention.

  The remote communication system 100 is a communication system for communicating with a remote place (remote place). The remote communication system 100 can communicate between a plurality of conference rooms. In this embodiment, as an example, it is assumed that communication is performed between two conference rooms A and B. The conference rooms A and B may be separated so as to require time for movement, or may be separate conference rooms provided in the same building. In other words, the expression “remote” in the present embodiment indicates that the physical space is different and does not define the distance or the length of time required for movement. In the remote communication system 100, an image processing apparatus 101a is installed in the conference room A as shown in FIG. 1A, and an image processing apparatus 101b is installed in the conference room B as shown in FIG. 1B. Yes. In this embodiment, the image processing apparatus 101a and the image processing apparatus 101b have the same configuration, and the configuration will be described below using the image processing apparatus 101a as an example. In addition, hereinafter, each component of the apparatus provided in the conference room A and an image displayed in the conference room A are denoted by “a” at the end, and each component of the apparatus provided in the conference room B and the conference A description will be given by attaching the symbol “b” to the end of the image or the like displayed in room B.

  In FIG. 1A, an image processing apparatus 101a includes a projector 102a and a depth sensor 104a. The image processing apparatus 101a is connected to a touch mat 107a, a camera 108a, and a display 109a.

  As shown in FIG. 2, the projector 102a includes a light source unit 201a, a projector barrel 202a, a reflection mirror 204a, and a projection mirror 205a. The light source unit 201a and the projector barrel portion 202a are connected via a bent portion 203a, and a reflection mirror 204a is disposed on the bent portion 203a. The light source unit 201a is a light source of the projector 102a and outputs light to the reflection mirror 204a. The projector barrel 202a is a projection optical system having a plurality of lenses. The reflection mirror 204a reflects the light output from the light source unit 201a to the projection mirror 205a, and the projection mirror 205a outputs the light reflected by the projection mirror 205a to the touch mat 107a. Thereby, the projector 102a projects each image on the touch mat 107a. Hereinafter, an area where the projector 102a projects an image on the touch mat 107a is defined as a projection area 103a (display area). Further, as the size of the projection area 103a, the width is defined as “screen_width” and the height is defined as “screen_height”. In the present embodiment, calibration processing relating to the projection area 103a of the conference room A and the projection area 103b of the conference room B is executed in advance. Thereby, the size of the projection area 103a in the conference room A and the position of the projection area 103a with respect to the touch mat 107a are the same as the size of the projection area 103b in the conference room B and the position of the projection area 103b with respect to the touch mat 107b.

  The depth sensor 104a extracts the area of the finger or hand of the user who points the image 105a such as the electronic material projected on the touch mat 107a with a finger or the like. Thus, when the users A and B in the conference room A point to the image 105a projected on the touch mat 107a, a pointing image indicating the action pointed to by the users A and B is generated. The generated pointing image is transmitted to the image processing apparatus 101b, and the image processing apparatus 101b projects the received pointing image on the touch mat 107b as a pointing image 106b. The depth sensor 104a can acquire subject distance information in a range from the touch mat 107a to the height d1 in FIG. The depth sensor 104a captures the touch mat 107a and generates an RGB image.

  The touch mat 107a is an input interface for accepting a touch operation by a user, and detects coordinates indicating a position where the touch operation is performed. The camera 108a captures the periphery of the touch mat 107a, and the display 109a displays the video of the user C in the conference room B.

  In the present embodiment, similarly to the image processing apparatus 101a, the image processing apparatus 101b also acquires distance information of the subject in the range from the touch mat 107b to the height d1 in FIG. The region of the finger or hand of user C pointing to is extracted. The image processing apparatus 101b generates a pointing image based on the extracted result, and transmits the pointing image to the image processing apparatus 101a. As a result, the actions indicating the images 105a and 105b can be notified to each other between the conference rooms A and B.

  FIG. 4 is a block diagram showing an outline of the hardware configuration of the image processing apparatus 101a of FIG.

  4, the image processing apparatus 101a includes a control unit 400a in addition to the projector 102a and the depth sensor 104a in FIG. 1, and the control unit 400a is connected to each of the projector 102a and the depth sensor 104a. The control unit 400a includes a CPU 401a, RAM 402a, ROM 403a, HDD 404a, camera I / F 405a, display controller 406a, audio controller 407a, USB controller 410a, and network I / F 411a. The CPU 401a, RAM 402a, ROM 403a, HDD 404a, camera I / F 405a, display controller 406a, audio controller 407a, USB controller 410a, and network I / F 411a are connected to each other via a system bus 419a. The depth sensor 104a includes a controller 415a, an infrared projection unit 416a, an infrared camera 417a, and an RGB camera 418a. The controller 415a is connected to each of the infrared projection unit 416a, the infrared camera 417a, and the RGB camera 418a.

  The control unit 400a comprehensively controls the entire image processing apparatus 101a. The CPU 401a executes each program by executing a program stored in the ROM 403a and the HDD 404a. The RAM 402a is a volatile memory and is used as a work area for the CPU 401a. The RAM 402a is used as a temporary storage area for each data, and stores, for example, a pointing image 906 and layout information described later. The ROM 403a is a non-volatile memory and stores a boot program of the image processing apparatus 101a. The HDD 404a stores the control program and each data of the image processing apparatus 101a. For example, the HDD 404a stores installation position information of the image processing apparatus 101a and installation position information of the display 109a. The camera I / F 405a performs data communication with the depth sensor 104a and the camera 108a. For example, the camera I / F 405a acquires the video data of the conference room A captured by the camera 108a from the camera 108a. The video data of the conference room A is transmitted to the image processing apparatus 101b via the network I / F 411a.

  The display controller 406a performs data communication with the projector 102a and the display 109a. The display controller 406a controls image projection processing by the projector 102a and image display control by the display 109a. For example, the display controller 406a performs control to project the setting menu 501a in FIG. 5A onto the projection area 103a of the touch mat 107a. The setting menu 501a is a circular image centered on a position pressed by the user for a predetermined period in the projection area 103a of the touch mat 107a, for example, the touch position 502a. The setting menu 501a includes at least a start button 503a. The start button 503a is an operation button for instructing execution of remote communication communication. When the start button 503a is selected by the user, a setting menu 504a shown in FIG. 5B is projected onto the projection area 103a. The setting menu 504a includes a connection list 505a, a connection button 506a, and a cancel button 507a. In the connection list 505a, communication destination candidates for remote communication communication are displayed. The connection button 506a is an operation button for instructing the start of remote communication communication, and the cancel button 507a is an operation button for instructing cancellation of settings related to remote communication communication. For example, when the conference room B is selected in the connection list 505a and the connection button 506a is selected, the image processing apparatus 101a transmits a remote communication communication execution request notification to the image processing apparatus 101b arranged in the conference room B. . When the acceptance of the execution request notification is completed, the remote communication communication can be executed between the image processing apparatus 101a and the image processing apparatus 101b. As a result, the image processing apparatus 101a can project an image 508a such as electronic material shared with the image processing apparatus 101b onto the projection area 103a as shown in FIG. 5C. Further, when the projection area 103a of the touch mat 107a is pressed by the user for a predetermined period while projecting the image 508a, the image processing apparatus 101a projects the setting menu 509a onto the projection area 103a. The setting menu 509a is a circular image centered on the touch position 510a pressed by the user, and includes a file button 511a and an end button 512a. The file button 511a is an operation button for selecting electronic materials to be projected onto the projection area 103a, and the end button 512a is an operation button for instructing the end of execution of the remote communication communication.

  The audio controller 407a performs data communication with the microphone 408a and the speaker 409a connected to the image processing apparatus 101a. For example, the audio controller 407a converts audio received by the microphone 408a into audio data. The audio data is transmitted to the image processing apparatus 101b through the network I / F 411a. Further, the audio controller 407a controls the speaker 409a to output the voice uttered by the user C in the conference room B to the conference room A. The USB controller 410a performs data communication with the connected touch mat 107a and the external memory 412a. For example, the USB controller 410a acquires coordinate information indicating the position of the touch mat 107a touched by the user with a pen or a finger from the touch mat 107a. The USB controller 410a acquires image data from an external memory 412a such as a connected USB memory or SD card. The network I / F 411a performs data communication with the file server 414 and the image processing apparatus 101b connected to the network 413.

  The controller 415a controls the infrared projection unit 416a and the infrared camera 417a to acquire 8-bit 1-channel distance image data indicating the distance from the image processing apparatus 101a to the object. The controller 415a controls the RGB camera 418a to acquire 8-bit 3-channel RGB image data. The infrared projection unit 416a projects an invisible infrared three-dimensional measurement pattern onto an object. The infrared camera 417a reads the three-dimensional measurement pattern projected on the object. The RGB camera 418a captures visible light. The distance image data and the RGB image data are stored in the HDD 404a.

  Next, remote communication communication executed by the remote communication system 100 will be described. Hereinafter, as an example, processing of the image processing apparatus 101a in remote communication communication will be described. Note that the image processing apparatus 101b can also perform the same processing.

  FIG. 6 is a flowchart showing a procedure of remote communication control processing executed by the image processing apparatus 101a of FIG.

  The processing in FIG. 6 is performed by the CPU 401a executing a program stored in the ROM 403a and the HDD 404a. 6 executes remote communication communication based on either the operation of the setting menu 504a in the image processing apparatus 101a or the remote communication communication execution request notification transmitted from another apparatus including the image processing apparatus 101b. Assumes the case.

  In FIG. 6, first, the CPU 401a determines whether the remote communication communication start instruction means is an operation of the setting menu 504a or a remote communication communication execution request notification transmitted from another device (step S601).

  As a result of the determination in step S601, when the start instruction means is an operation of the setting menu 504a, the CPU 401a transmits a remote communication communication execution request notification to the communication destination selected from the connection list 505a, for example, the image processing apparatus 101b ( Step S602). The remote communication communication execution request notification includes information necessary for remote communication communication, information for specifying electronic materials to be shared, and the like. Next, the CPU 401a determines whether or not a remote communication communication acceptance completion notification has been received from the image processing apparatus 101b (step S603).

  As a result of the determination in step S603, when the reception completion notification is received, the CPU 401a performs the processing after step S607 described later. On the other hand, when the reception completion notification is not received as a result of the determination in step S603, the CPU 401a determines whether or not a remote communication communication execution request cancellation notification is received from the image processing apparatus 101b (step S604).

  When the execution request cancellation notification is received as a result of the determination in step S604, the CPU 401a ends this process. On the other hand, as a result of the determination in step S604, when the execution request cancellation notification is not received, the CPU 401a determines whether or not the cancel button 507a is selected in the setting menu 504a (step S605).

  As a result of the determination in step S605, when the cancel button 507a is selected, the CPU 401a ends this process. On the other hand, as a result of the determination in step S605, when the cancel button 507a is not selected, the CPU 401a returns to the process in step S603.

  As a result of the determination in step S601, when the start instruction means is a remote communication communication execution request notification transmitted from another device, the CPU 401a determines electronic materials to be shared with the image processing apparatus 101b based on the execution request notification. (Step S606). Next, the CPU 401a executes initialization processing for each module for executing remote communication communication (step S607). When the execution of the initialization process is completed, the CPU 401a executes a pointing image transmission process shown in FIG. 7 and a pointing image reception process shown in FIG. The pointing image transmission process of FIG. 7 and the pointing image reception process of FIG. 10 to be described later are repeatedly executed until the end of the remote communication communication is instructed by the user. When the projection area 103a is pressed for a predetermined period, the CPU 401a projects a setting menu 509a onto the projection area 103a. When the end button 512a is selected (YES in step S608), the CPU 401a executes remote communication communication end processing (step S609), and ends this processing.

  Next, transmission and reception of a pointing image in remote communication communication will be described. As an example, a case where the image processing apparatus 101a transmits a pointing image to the image processing apparatus 101b will be described.

  FIG. 7 is a flowchart showing the procedure of the pointing image transmission process executed by the image processing apparatus 101a of FIG.

  The processing in FIG. 7 is performed by the CPU 401a executing a program stored in the ROM 403a and the HDD 404a.

  In FIG. 7, first, when the CPU 401a detects the pointing of the image 105a by the users A and B in the conference room A, the CPU 401a executes the pointing image generation process of FIG. 8 (step S701). As a result, the CPU 401a generates a pointing image indicating an action in which the users A and B in the conference room A point to the image 105a, and layout information indicating the display position of the pointing image and the like. Next, the CPU 401a transmits the pointing image and layout information to the image processing apparatus 101b (step S702). In step S702, when the pointing of the image 105a is not detected in the conference room A, the CPU 401a transmits information indicating that the pointing of the image 105a is not detected to the image processing apparatus 101b. Next, the CPU 401a determines whether an instruction to end the pointing image transmission process has been received (step S703).

  As a result of the determination in step S703, when the instruction to end the pointing image generation process is not accepted, the CPU 401a returns to the process in step S701. On the other hand, as a result of the determination in step S703, when an instruction to end the pointing image generation process is received, the CPU 401a performs the process from step S608 in FIG. 6 described above.

  FIG. 8 is a flowchart showing the procedure of the pointing image generation process in step S701 of FIG.

  In FIG. 8, first, the CPU 401a acquires an RGB image and a distance image from the depth sensor 104a (steps S801 and S802), and performs coordinate conversion of the distance image (step S803). In the depth sensor 104a, since the infrared camera 417a that captures the distance image is provided at a different position from the RGB camera 418a that captures the RGB image, the CPU 401a converts the distance image into the coordinate system of the RGB image in the process of step S803. In the present embodiment, calibration processing relating to the infrared camera 417a and the RGB camera 418a is executed in advance. Thereby, each positional information of the infrared camera 417a and the RGB camera 418a and internal parameters used for the coordinate conversion are derived, and a reference image is generated. The reference image is a distance image of only the touch mat 107a that does not include the hands of the users A and B.

  Next, the CPU 401a generates a mask image based on the difference between the reference image generated in advance and the distance image acquired in step S802 (step S804). In step S804, for example, a distance image including information on the hand of user A pointing to the image 105a in the conference room A is generated as a mask image. Next, the CPU 401a performs image processing for noise removal and smoothing on the generated mask image (step S805).

  Next, the CPU 401a generates an intermediate image including color information based on the mask image subjected to the image processing and the RGB image acquired in step S801 (step S806). Next, the CPU 401a performs a transparency process on the intermediate image (step S807). In step S807, the multi-valued image 900a in FIG. 9A is generated in which the portion having the pixel value “255” is made non-transparent and the portion having the pixel value “0” is made transparent in the intermediate image. In the multi-valued image 900a, the area 901a indicating the hand of the user A is translucent, and the other areas are transparent. Next, the CPU 401a projects and converts the multi-valued image 900a into a projection coordinate system that is the coordinate system of the projection area 103a, and generates a converted image 902a in FIG. 9B (step S808). Specifically, in step S808, the vertex (L1, M1) of the projection region 103a included in the multi-valued image 900a is converted to (0, 0), and the vertex (L2, M2) is converted to (0, screen_height). . The vertex (L3, M3) is converted to (screen_width, screen_height), and the vertex (L4, M4) is converted to (screen_width, 0). Each parameter used for projective transformation is derived by a prior calibration process.

  Next, the CPU 401a extracts the circumscribed rectangular area 904a of the area 903a indicating the user A's hand in the converted image 902a, and acquires the coordinates of the circumscribed rectangular area 904a (step S809). Next, the CPU 401a acquires the coordinates of the indicated position in the converted image 902a (step S810). The pointing position is the tip of the finger pointed by the user A in the converted image 902a. In step S810, the CPU 401a identifies a side 905a that coincides with the end of the converted image 902a from the four sides of the circumscribed rectangular region 904a and that is most in contact with non-transparent pixels. The CPU 401a points to the position 907a where the non-transparent pixel is in contact with the side 906a facing the side 905a, and acquires the coordinates of the pointed position.

  Next, the CPU 401a trims the circumscribed rectangular area 904a from the converted image 902a, and generates the pointing image 908a of FIG. 9C having directionality (step S811). Hereinafter, the fingertip side of the user A in the pointing image 908a is defined as the fingertip side of the pointing image 908a, and the wrist side of the user A in the pointing image 908a is defined as the wrist side of the pointing image 908a. Further, a straight line connecting the pointing position 907a and the wrist side of the pointing image 908a is defined as a pointing line 909a. Further, the direction indicated by the user A is defined as the pointing direction, and the direction is indicated by an arrow of the line 909a. Next, the CPU 401a associates the pointing image 908a with the layout information and stores them in the RAM 402a (step S811). The layout information includes the coordinates of the circumscribed rectangular area 904a acquired in step S809, the coordinates of the pointing position acquired in step S810, and the like. Thereafter, the CPU 401a performs the processing after step S702 described above. In the present embodiment, when a plurality of indications are detected, the processing of FIG. 8 is executed for each indication.

  FIG. 10 is a flowchart showing the procedure of the pointing image reception process executed by the image processing apparatus 101b of FIG.

  The processing in FIG. 10 is performed by the CPU 401b executing a program stored in the ROM 403b and the HDD 404b.

  In FIG. 10, the CPU 401b determines whether a pointing image and layout information are received from the image processing apparatus 101a (step S1001).

  As a result of the determination in step S1001, for example, when receiving a pointing image 908a and layout information from the image processing apparatus 101a, the CPU 401b performs layout information update processing in FIG. 11 described later (step S1002). As a result, the CPU 401b determines a rotation angle for rotating the acquired pointing image 908a, and updates the layout information based on the rotation angle. Next, the CPU 401b projects the pointing image on the projection area 103b based on the pointing image 908a and the updated layout information (step S1003). Next, the CPU 401b determines whether or not the end of execution of the pointing image reception process has been instructed (step S1004).

  As a result of the determination in step S1004, when the end of execution of the pointing image reception process is not instructed, the CPU 401b returns to the process in step S1001. On the other hand, when the end of execution of the pointing image reception process is instructed, the CPU 401b performs the processes after step S608 described above.

  FIG. 11 is a flowchart showing the procedure of the layout information update process in step S1002 of FIG.

  In FIG. 11, first, the CPU 401b obtains information about the installation position of the image processing apparatus 101b with respect to the projection area 103b (hereinafter simply referred to as “the installation position of the image processing apparatus 101b”) from the HDD 404b (step S1101). Next, the CPU 401b determines whether the wrist side of the pointing image projected on the projection area 103b is the installation side of the image processing apparatus 101b based on the acquired pointing image 908a (step S1102). Specifically, either the pointing line of the pointing image or an extension line extending in the direction opposite to the pointing direction (hereinafter referred to as “pointing extension line”) is set in the projection region 103b. When intersecting the side edge, the CPU 401b determines that the wrist side of the pointing image is the installation side of the image processing apparatus 101b. For example, when the pointing image is the pointing image 1201b of FIG. 12A, the pointing line 1202b of the pointing image 1201b intersects the side 1203b on the installation side of the image processing apparatus 101b in the projection area 103b. It is determined that the wrist side of the pointing image is the installation side of the image processing apparatus 101b.

  On the other hand, when neither the pointing line nor the pointing extension line of the pointing image intersects the side on the installation side of the image processing apparatus 101b in the projection area 103b, the CPU 401b indicates that the wrist side of the pointing image is not the installation side of the image processing apparatus 101b. Is determined. For example, when the pointing image is the pointing image 1204b in FIG. 12A, the pointing line 1205b of the pointing image 1204b does not intersect the side 1203b, so the CPU 401b has the wrist side of the pointing image on the installation side of the image processing apparatus 101b. It is determined that it is not.

  As a result of the determination in step S1102, when the wrist side of the pointing image is the installation side of the image processing apparatus 101b, the CPU 401b ends this process. On the other hand, if the result of determination in step S1102 is that the wrist side of the pointing image is not the installation side of the image processing apparatus 101b, the CPU 401b determines the rotation angle of the pointing image (step S1103). For example, when the pointing image 1204b is rotated around the pointing position of the pointing image 1204b, the CPU 401b determines the rotation angle at which the extended line indicated by the pointing image 1204b intersects the side on the installation side of the image processing apparatus 101b. Next, the CPU 401b updates the layout information based on the rotation angle (step S1104), and stores the updated layout information in the RAM 402b. As a result, the pointing image 1207b of FIG. 12C in which the wrist side of the user A is the image processing apparatus 101b side is projected onto the projection area 103b. That is, the image processing apparatus 101b displays the pointing image 1207b from the side where the user C does not exist, specifically from the installation side of the image processing apparatus 101b. After that, the CPU 401b performs the processing after step S1003 described above.

  10 and 11 described above, the pointing image 1207b is displayed from the side where the user C of the image processing apparatus 101b does not exist in the projection area 103b of the image processing apparatus 101b. That is, regardless of the positions of the users A and B in the conference room A, the pointing image 1207b is not displayed at the position of the user C in the conference room B. Thereby, in the conference room B, it is possible to avoid an unnatural display such as a hand jumping out from the user C.

  Further, in the above-described processing of FIGS. 10 and 11, the pointing image 1207b is displayed from the installation side of the image processing apparatus 101b in the projection region 103b of the image processing apparatus 101b. Thereby, at least the pointing image 1207b can be prevented from being superimposed on the user C and displayed.

  As described above, the present invention has been described using the above-described embodiment, but the present invention is not limited to the above-described embodiment.

  For example, it may be a case where the user C is confirming the projection region 103b from the front side of the image processing apparatus 101b as shown in FIG. Even in this case, the same effect as the above-described embodiment can be obtained.

  In the above-described embodiment, the pointing image 1207b may be displayed from the upper end side in the vertical direction of the image 105b in the projection region 103b (hereinafter simply referred to as “the upper end side of the image 105b”).

  FIG. 13 is a flowchart showing the procedure of a first modification of the layout information update process of FIG.

  In FIG. 13, first, the CPU 401b acquires information on the upper end side of the image 105b projected on the projection area 103b from the HDD 404b (step S1301). Next, the CPU 401b determines whether the wrist side of the pointing image projected on the projection area 103b is the upper end side of the image 105b based on the acquired pointing image 908a (step S1302). Specifically, when any one of the pointing line and the pointing extension line of the pointing image intersects the upper edge side of the image 105b in the projection area 103b, the CPU 401b determines that the wrist side of the pointing image is the upper edge side of the image 105b. Determine. For example, when the pointing image is the pointing image 1401b of FIG. 14A, the pointing line 1402b of the pointing image 1401b crosses the side 1403b on the upper end side of the image 105b in the projection region 103b, and thus the CPU 401b It is determined that the wrist side is the upper end side of the image 105b.

  On the other hand, when neither the pointing line nor the pointing extension line intersects the upper edge side of the image 105b in the projection area 103b, the CPU 401b determines that the wrist side of the pointing image is not the upper edge side of the image 105b. For example, when the pointing image is the pointing image 1404b of FIG. 14A, the pointing line 1404b of the pointing image 1404b does not intersect the side 1403b, so the CPU 401b does not have to point the wrist side of the pointing image to the upper end side of the image 105b. Determine.

  As a result of the determination in step S1302, when the wrist side of the pointing image is the upper end side of the image 105b, the CPU 401b ends this process. On the other hand, as a result of the determination in step S1302, when the wrist side of the pointing image is not the upper end side of the image 105b, the CPU 401b determines the rotation angle of the pointing image (step S1303). For example, when the pointing image 1404b is rotated around the pointing position of the pointing image 1404b, the CPU 401b determines the rotation angle at which the extended line indicated by the pointing image 1404b intersects the upper edge side of the image 105b. Next, the CPU 401b updates the layout information based on the rotation angle (step S1304), and stores the updated layout information in the RAM 402b. Accordingly, a pointing image 1407b in which the wrist side of the user A is the upper end side of the image 105b is projected onto the projection area 103b. That is, the image processing apparatus 101b displays the pointing image 1407b from the upper end side of the image 105b. Thereafter, the CPU 401b ends this process.

  In the process of FIG. 13 described above, the pointing image 1407b is displayed from the upper end side of the image 105b in the projection region 103b. Normally, since the user confirms the image 105b projected on the projection area 103b from a position where the user can easily view the image 105b, the possibility that the user C exists on the upper end side of the image 105 which is a position where the image 105b is difficult to view is extremely low. On the other hand, in the present embodiment, the pointing image 1407b is displayed from the upper end side of the image 105b in the projection region 103b. Thereby, the situation where the pointing image 1407b is displayed superimposed on the user C can be reliably reduced.

  In the above-described embodiment, a position where the user C is not present in the conference room B may be estimated using the camera 108b or the microphone 408b, and a pointing image may be displayed from the estimated side.

  FIG. 15 is a flowchart showing the procedure of the second modification of the layout information update process of FIG.

  In FIG. 15, first, the CPU 401b estimates a side where no user exists among the four sides of the projection area 103b (step S1501). In the following, in step S1501, the side estimated to have no user in the projection area 103b is defined as the unmanned estimated side, and the unmanned estimated side is defined as the unmanned estimated side. For example, in step S1501, the CPU 401b analyzes a photographed image photographed by the camera 108b, and detects a face area from the photographed image based on a method using the Haar Like feature. The CPU 401b sets the side where the face area is not detected among the four sides of the projection area 103b as the unmanned estimated side. In the present embodiment, the relationship between the captured image and each side of the projection area 103b is set in advance by prior calibration. Further, the CPU 401b detects an utterance around the projection area 103b and an utterance direction that is a direction from the utterance source to the projection area 103b based on a method using a microphone array. The CPU 401b starts detecting the utterance and the utterance direction in response to the start of the remote communication communication, and unattended estimation is performed on the side of the four sides of the projection area 103b where no utterance is detected or the number of utterance detection is less than the preset number. It is the side. In the present embodiment, the relationship between the utterance direction and each side of the projection region 103b is set in advance by prior calibration.

  Next, based on the acquired pointing image 908a, the CPU 401b determines whether or not the wrist side of the pointing image projected on the projection area 103b is the unmanned estimated side (step S1502). Specifically, when any of the pointing line and the pointing extension line of the pointing image intersects the unmanned estimated side, the CPU 401b determines that the wrist side of the pointing image is the unmanned estimated side. For example, when the pointing image is the pointing image 1601b of FIG. 16A, the pointing line 1602b of the pointing image 1601b intersects the unmanned estimated side 1603b, so the CPU 401b has the wrist side of the pointing image on the unmanned estimated side side. Determine that there is. On the other hand, when neither the pointing line nor the pointing extension line intersects the unmanned estimated side, the CPU 401b determines that the wrist side of the pointing image is not the unmanned estimated side. For example, if the pointing image is the pointing image 1604b of FIG. 16A, the pointing line 1605b of the pointing image 1604b does not intersect the unmanned estimated side 1603b, so the CPU 401b does not have the wrist side of the pointing image on the unmanned estimated side side. Is determined.

  As a result of the determination in step S1502, when the wrist side of the pointing image is the unmanned estimated side, the CPU 401b ends this process. On the other hand, if the result of determination in step S1502 is that the wrist side of the pointing image is not the unmanned estimated side, the CPU 401b determines the rotation angle of the pointing image (step S1503). For example, when the pointing image 1604b is rotated around the pointing position of the pointing image 1604b, the CPU 401b determines a rotation angle at which the extension line of the pointing image 1604b intersects the unmanned estimated side. Next, the CPU 401b updates the layout information based on the rotation angle (step S1504), and stores the updated layout information in the RAM 402b. Thereby, a pointing image 1607b in which the wrist side of the user A is the unmanned estimated side is projected onto the projection area 103b. That is, the image processing apparatus 101b displays the pointing image 1607b from the unmanned estimated side. Thereafter, the CPU 401b ends this process.

  In the process of FIG. 15 described above, the pointing image 1607b is displayed from the unmanned estimated side. Thereby, the situation where the pointing image 1407b is displayed superimposed on the user C can be more reliably reduced.

  In the above-described embodiment, a pointing image may be displayed from the display 109b side.

  FIG. 17 is a flowchart showing the procedure of the third modification of the layout information update process of FIG.

  In FIG. 17, first, the CPU 401b obtains information regarding the installation position of the display 109b on which the images of the communication destination users A and B are displayed from the HDD 404b (step S1701). Next, the CPU 401b determines whether or not the wrist side of the pointing image projected on the projection area 103b is the installation side of the display 109b based on the acquired pointing image 908a (step S1702). Specifically, when either the pointing line or the pointing extension line of the pointing image intersects the side of the projection area 103b where the display 109b is installed, the CPU 401b determines that the wrist side of the pointing image is the installation side of the display 109b. Determine that there is. Hereinafter, the side of the projection area 103b on which the display 109b is installed is defined as the display installation side. On the other hand, when neither the pointing line nor the pointing extension line intersects the display installation side, the CPU 401b determines that the wrist side of the pointing image is not the display 109b installation side.

  As a result of the determination in step S1702, when the wrist side of the pointing image is the installation side of the display 109b, the CPU 401b ends this process. On the other hand, if the result of determination in step S1702 is that the wrist side of the pointing image is not the display 109b installation side, the CPU 401b determines the rotation angle of the pointing image (step S1703). For example, when the CPU 401b rotates a pointing image where the wrist of the user A is not the display 109b on the pointing image pointing position, the CPU 401b sets the rotation angle at which the extension line of the pointing image intersects the display setting side. decide. Next, the CPU 401b updates the layout information based on the determined rotation angle (step S1704), and stores the updated layout information in the RAM 402b. Thus, a pointing image 1801b in which the wrist side of the user A is the installation side of the display 109b is projected onto the projection area 103b. That is, the image processing apparatus 101b displays the pointing image 1801b from the installation side of the display 109b. Thereafter, the CPU 401b ends this process.

  In the processing of FIG. 17 described above, the pointing image 1801b is displayed from the installation side of the display 109b in the projection region 103b. Thereby, in the conference room B, the pointing images of the users A and B in the conference room A can be displayed from the side where the users A and B are displayed, and thus natural remote communication communication can be realized.

  Furthermore, in the above-described embodiment, the layout information may be updated before the image processing apparatus 101a transmits the layout information.

  FIG. 19A is a flowchart illustrating a procedure of a modification of the pointing image reception process of FIG.

  The process in FIG. 19A is performed by the CPU 401b executing a program stored in the ROM 403b and the HDD 404b.

  19A, the CPU 401b transmits a parameter for determining the rotation angle of the pointing image to the image processing apparatus 101a (step S1901). That is, in the present embodiment, the image processing apparatus 101b transmits the parameters to the image processing apparatus 101a before receiving the pointing image and the layout information from the image processing apparatus 101a. Examples of the parameters include information regarding the installation position of the image processing apparatus 101b with respect to the projection area 103b, information indicating the upper edge side of the image 105b in the projection area 103b, information indicating the unmanned estimated edge side in the projection area 103b, and the display 109b Information regarding the installation position of the image processing apparatus 101a is transmitted to the image processing apparatus 101a. Next, the process of step S1001 described above is executed.

  As a result of the determination in step S1001, when the pointing image and the layout information are not received from the image processing apparatus 101a, the CPU 401b performs the processes after step S1004 described above. On the other hand, as a result of the determination in step S1001, when the pointing image and the layout information of the pointing image are received from the image processing apparatus 101a, the CPU 401b performs the processing from step S1003 described above.

  FIG. 19B is a flowchart showing the procedure of a modification of the pointing image transmission process of FIG.

  The process in FIG. 19B is performed by the CPU 401a executing a program stored in the ROM 403a and the HDD 404a.

  In FIG. 19B, first, the CPU 401a acquires the parameters from the image processing apparatus 101b (step S1902). Next, the CPU 401a performs the process of step S701 described above and executes a layout information update process (step S1903). In step S1903, the layout information update process in any one of FIGS. 20A to 20D is executed. For example, in the process of FIG. 20A, the CPU 401a executes the processes of steps S1102 to S1104 described above, and the layout information is displayed so that the pointing image is displayed from the installation side of the image processing apparatus 101b in the image processing apparatus 101b. Update. In the process of FIG. 20B, the CPU 401a executes the processes of steps S1302 to S1304 described above, and updates the layout information so that the pointing image is displayed from the upper end side of the image 105b in the image processing apparatus 101b. . Further, in the process of FIG. 20C, the CPU 401a executes the processes of steps S1502 to S1504 described above, and updates the layout information so that the pointing image is displayed from the unmanned estimated side in the image processing apparatus 101b. In the process of FIG. 20D, the CPU 401a executes the processes of steps S1702 to S1704 described above, and updates the layout information so that the pointing image is displayed from the installation side of the display 109b in the image processing apparatus 101b. Next, the CPU 401a performs the processing after step S702 described above.

  In the processes of FIGS. 19A, 19B, and 20A to 20D described above, the same effects as those of the present embodiment described above can be obtained.

  In addition, in the present embodiment described above, the parameter transmission process that is extremely unlikely to be changed during remote communication communication, such as the installation position information of the image processing apparatus 101b and the installation position information of the display 109b, The image processing apparatus 101a does not need to perform the process a plurality of times, and preferably holds the received parameter. As a result, the execution of parameter communication processing can be reduced, and the load of parameter communication processing can be reduced.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read the program. It can also be realized by processing to be executed. The present invention can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Remote communication system 101a, 101b Image processing apparatus 102a Projector 103a, 103b Projection area 105a, 105b Image 108a, 108b Camera 408a, 408b Microphone 908a, 1207b, 1407b, 1607b, 1801b Pointing image

Claims (12)

A display control system that includes an image processing device and another image processing device, and displays a shared image in each display area of the image processing device and the other image processing device,
Display control means for performing control to display a pointing image indicating an action in which the user of the other image processing device points to the displayed image in a display area of the image processing device;
The pointing image has directionality,
The display control system, wherein the display control means displays the pointing image from the side where the user of the image processing apparatus does not exist in the display area of the image processing apparatus.   The display control system according to claim 1, wherein the display control unit displays the pointing image from a side where the image processing device is arranged in a display area of the image processing device.   The display control system according to claim 1, wherein the display control unit displays the pointing image from an upper end side of the displayed image in a vertical direction of the displayed image. The image processing apparatus further comprises detection means for detecting users around the image processing apparatus,
The display control system according to claim 1, wherein the display control unit displays the pointing image from a side where the user is not detected in a display area of the image processing apparatus. A display unit for displaying a user of the other image processing apparatus;
The display control system according to claim 1, wherein the display control unit displays the pointing image from a side where the display unit is arranged in a display area of the image processing apparatus. An image processing apparatus that displays an image shared with another image processing apparatus in a display area,
Display control means for performing control to display a pointing image indicating an action in which the user of the other image processing apparatus points to the displayed image on the display area;
The pointing image has directionality,
The display control means displays the pointing image from the side where the user of the image processing apparatus does not exist in the display area.   The image processing apparatus according to claim 6, wherein the display control unit displays the pointing image from a side where the image processing apparatus is arranged in the display area.   The image processing apparatus according to claim 6, wherein the display control unit displays the pointing image from an upper end side of the displayed image in a vertical direction of the displayed image. The image processing apparatus further comprises detection means for detecting users around the image processing apparatus,
The image processing apparatus according to claim 6, wherein the display control unit displays the pointing image from a side where the user is not detected in the display area. A display unit for displaying a user of the other image processing apparatus;
The image processing apparatus according to claim 6, wherein the display control unit displays the pointing image from the side where the display unit is arranged in the display area. A display control method comprising an image processing device and another image processing device, and displaying a shared image in each display area of the image processing device and the other image processing device,
A display control step for performing control to display a pointing image indicating an action in which the user of the other image processing device points to the displayed image in a display area of the image processing device;
The pointing image has directionality,
In the display control step, the pointing image is displayed from the side where the user of the image processing apparatus does not exist in the display area of the image processing apparatus. A program comprising an image processing device and another image processing device, causing a computer to execute a display control method for displaying a shared image in each display area of the image processing device and the other image processing device,
The display control method includes:
A display control step for performing control to display a pointing image indicating an action in which the user of the other image processing device points to the displayed image in a display area of the image processing device;
The pointing image has directionality,
The display control step displays the pointing image from the side where the user of the image processing apparatus does not exist in the display area of the image processing apparatus.

Images