JP2012175456A - Image processing device, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device, an image processing method, and an image processing program, capable of performing proper processing depending on a situation at the time of change when an imaging range of imaging means changes.SOLUTION: A PC acquires movement information indicating a movement state of an imaging range of a camera provided in a teleconference terminal (S3 and S8). Whether or not the imaging range of the camera moves is determined by the acquired movement information (S4, S5, and S11). When determined that the imaging range moves and when the movement is terminated (S12:YES), the PC determines whether or not a user moves the camera intentionally (S14 and S15). When determined that the user does not move the camera intentionally (S15:NO), the PC determines whether or not a transmission image range that is an imaging range of image data to be transmitted to a communication device on the other site can be restored. When determined that the transmission image range can be restored, the transmission image range is restored. When determined that the transmission image range cannot be restored, data of an under-modification notification image is transmitted instead of the data of a taken image (S16).

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program for processing image data to be transmitted to a communication apparatus at another base in order to execute a remote conference.

  2. Description of the Related Art Conventionally, a technique for restoring an imaging range to a range before the change occurs when the imaging range of the imaging unit is changed is known. For example, the image processing system disclosed in Patent Document 1 detects a change (positional deviation) in the imaging range using a motion vector calculated for each block. When a change in imaging is detected, the imaging range is restored by returning the camera to a position where the original normal scene is imaged.

JP 2002-77708 A

  Even when the imaging range changes against the user's intention, the imaging range cannot always be restored. In the prior art, it was not possible to perform appropriate processing depending on whether or not the imaging range can be restored. Further, in the conventional technique, there is a case where processing for restoring the imaging range is performed even when the user intentionally changes the imaging range. That is, there is a problem in that appropriate processing cannot be performed depending on the situation when the imaging range changes.

  An object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing program capable of performing appropriate processing in accordance with the situation when the imaging range of the imaging unit changes. To do.

  An image processing apparatus according to a first aspect of the present invention is an image processing apparatus that processes image data to be transmitted to communication devices arranged at other bases in order to execute a remote conference by users at a plurality of bases. An image data acquisition unit that acquires image data of an image captured by an imaging unit that captures an image, a movement information acquisition unit that acquires movement information indicating a movement state of an imaging range of the imaging unit, and the movement information acquisition The movement information acquired by the movement information acquired by the movement determination unit that determines whether or not the imaging range of the imaging unit has moved based on the movement information acquired by the unit, and the movement information acquisition when the movement determination unit determines that the imaging range has moved. The user deliberately sets the imaging range by using at least one of the movement information acquired by the means and the operation state information indicating the operation state of the imaging means. A deliberate determination means for determining whether or not it has been moved, and a transmission image range that is an imaging range of image data to be transmitted to the communication device when it is determined by the intention determination means that it has not been intentionally moved. When it is determined that the transmission image range can be restored by the restoration judgment unit, and the restoration judgment unit judges whether or not the imaging range can be restored by using the image data acquired by the image data acquisition unit A restoration unit that restores the transmission image range to an imaging range before movement, and a process that changes an image displayed on a display device when the restoration determination unit determines that the transmission image range cannot be restored, and Processing means for performing at least one of processes for changing a frame rate of image data transmitted to the communication apparatus.

  According to the image processing apparatus according to the first aspect, when it is determined that the movement of the imaging range of the imaging unit is due to an accident and it is determined that the transmission image range can be restored, the transmission image range becomes the imaging range before the movement. Restored. Therefore, the user's labor is reduced. In addition, when the transmission image range cannot be restored, the image processing device can perform at least one of processing for changing an image displayed on the display device and processing for changing the frame rate of image data to be transmitted. As a result, it is possible to notify the user that the imaging range has moved unintentionally, or to prevent transmission of image data that does not need to be transmitted. Further, when it is determined that the user has intentionally moved the imaging range of the imaging means, processing such as restoration of the transmission image range and change of image data is not performed. Therefore, the user can move the transmission image range as intended. Therefore, the image processing apparatus can perform appropriate processing according to the situation when the imaging range of the imaging unit changes.

  The intention determination unit may determine that the user has intentionally moved the imaging range when the vertical movement amount indicated by the movement information acquired by the movement information acquisition unit is equal to or greater than a first threshold. When the user intentionally moves with the image pickup means, the image pickup means is accompanied by a movement in the vertical direction. Therefore, the image processing apparatus can appropriately determine whether or not the movement of the imaging range is intentional based on the amount of movement of the imaging range in the vertical direction.

  The deliberate determination means has a vertical movement amount indicated by the movement information acquired by the movement information acquisition means that is less than the first threshold value, a horizontal movement amount that is less than a second threshold value, and the movement When the moving time of the imaging range indicated by the information is less than the third threshold, the user may determine that the imaging range is not intentionally moved. When the movement amount of the imaging range is small and the movement time is short, the imaging range often moves when the user or the like suddenly touches the imaging means. Therefore, the image processing apparatus can appropriately determine whether or not the movement of the imaging range is intentional based on the movement amount and the movement time of the imaging range.

  The restoring means is a specifying means for specifying the same reference imaged object in each of the image data before and after it is determined that the imaging range has been moved by the movement determining means among the image data acquired by the image data acquiring means. May be provided. The restoration unit is configured such that the position in the imaging range of the reference imaged object specified by the specifying unit in the image data after movement is the position of the reference imaged object specified by the specifying unit in the image data before movement. The transmission image range may be moved and restored so as to match. In this case, the image processing apparatus can restore the transmission image range easily and reliably. The specifying unit may specify a plurality of reference imaging objects.

  The intention determination unit may include a detection unit that detects a position of a speaker in the transmission image range. The intentional determination unit is configured such that when the position of the speaker detected by the detection unit after the movement is detected by the movement determination unit is within a predetermined region including the center point of the transmission image range, the user It may be determined that the imaging range is intentionally moved. When the imaging range of the imaging means moves, users at other bases can visually recognize the speaker if the position of the speaker is within a predetermined area including the center point of the transmission image range. In this case, the image processing apparatus can smoothly advance the remote conference by determining that the user has intentionally moved the imaging range.

  The intentional determination unit may determine that the user has not intentionally moved the imaging range when the operation state information indicates that the imaging unit has been operated. The user rarely intentionally performs both operation of the imaging means and movement of the imaging range. Therefore, the image processing apparatus can make an accurate determination by determining that the movement when the imaging unit is operated is an accident.

  The processing means may change the image data to be transmitted to the communication device to image data including an image for notifying that the imaging range of the imaging means has moved. In this case, the user can easily grasp that the imaging range has moved unintentionally, and can quickly take appropriate measures.

  An image processing method according to a second aspect of the present invention is performed by an image processing apparatus that processes image data to be transmitted to a communication apparatus arranged at another base in order to execute a remote conference by users at a plurality of bases. An image data acquisition step of acquiring image data of an image captured by an imaging unit that captures an image, and a movement information acquisition step of acquiring movement information indicating a movement state of an imaging range of the imaging unit And a movement determination step for determining whether or not the imaging range of the imaging means has moved according to the movement information acquired in the movement information acquisition step, and a case where it is determined that the imaging range has moved in the movement determination step In addition, the movement information acquired in the movement information acquisition step and the operation state information indicating the operation state with respect to the imaging unit are little. Is transmitted to the communication device when it is determined that the user has intentionally moved the imaging range and whether the user has intentionally moved in the intention determination step. A restoration determination step for determining whether the transmission image range, which is the imaging range of the image data to be restored, can be restored to the imaging range before the movement, using the image data acquired in the image data acquisition step, and the restoration determination step When it is determined that the transmission image range can be restored in step S2, it is determined that the transmission image range cannot be restored in the restoration step of restoring the transmission image range to the imaging range before movement, and the restoration determination step. A process for changing an image displayed on the display device, and a frame rate of image data transmitted to the communication device. And at least the process steps of performing any of the processing to be.

  According to the image processing method according to the second aspect, when it is determined that the movement of the imaging range of the imaging unit is due to an accident and it is determined that the transmission image range can be restored, the transmission image range becomes the imaging range before the movement. Restored. Therefore, the user's labor is reduced. In addition, when the transmission image range cannot be restored, the image processing device can perform at least one of processing for changing an image displayed on the display device and processing for changing the frame rate of image data to be transmitted. As a result, it is possible to notify the user that the imaging range has moved unintentionally, or to prevent transmission of image data that does not need to be transmitted. Further, when it is determined that the user has intentionally moved the imaging range of the imaging means, processing such as restoration of the transmission image range and change of image data is not performed. Therefore, the user can move the transmission image range as intended. Therefore, the image processing apparatus can perform appropriate processing according to the situation when the imaging range of the imaging unit changes.

  The image processing program according to the third aspect of the present invention is used in an image processing apparatus that processes image data to be transmitted to a communication apparatus arranged at another base in order to execute a remote conference by users at a plurality of bases. An image processing program for acquiring image data of an image captured by an imaging unit that captures an image, and a movement information acquisition step of acquiring movement information indicating a movement state of an imaging range of the imaging unit And a movement determination step for determining whether or not the imaging range of the imaging means has moved according to the movement information acquired in the movement information acquisition step, and a case where it is determined that the imaging range has moved in the movement determination step The operation information indicating the movement information acquired in the movement information acquisition step and the operation state with respect to the imaging unit. A deliberate determination step for determining whether or not the user has intentionally moved the imaging range using at least one of the information, and the communication device when it is determined that the user has not intentionally moved in the deliberate determination step A restoration determination step for determining whether or not a transmission image range that is an imaging range of image data to be transmitted to the imaging range before movement can be restored using the image data acquired in the image data acquisition step, and the restoration When it is determined that the transmission image range can be restored in the determination step, it is determined that the transmission image range cannot be restored in the restoration step of restoring the transmission image range to the imaging range before the movement, and the restoration judgment step. A process for changing an image displayed on the display device and a frame rate of image data transmitted to the communication device. And at least the process steps of performing any of the processing for changing the Mureto containing instructions to be executed by the controller of the image processing apparatus.

  According to the image processing program according to the third aspect, when it is determined that the movement of the imaging range of the imaging unit is due to an accident and it is determined that the transmission image range can be restored, the transmission image range becomes the imaging range before the movement. Restored. Therefore, the user's labor is reduced. In addition, when the transmission image range cannot be restored, the image processing device can perform at least one of processing for changing an image displayed on the display device and processing for changing the frame rate of image data to be transmitted. As a result, it is possible to notify the user that the imaging range has moved unintentionally, or to prevent transmission of image data that does not need to be transmitted. Further, when it is determined that the user has intentionally moved the imaging range of the imaging means, processing such as restoration of the transmission image range and change of image data is not performed. Therefore, the user can move the transmission image range as intended. Therefore, the image processing apparatus can perform appropriate processing according to the situation when the imaging range of the imaging unit changes.

2 is a configuration diagram of a PC 1 and a video conference terminal 3. FIG. 2 is a block diagram illustrating an electrical configuration of a PC 1 and a video conference terminal 3. FIG. It is explanatory drawing for demonstrating the acquisition image range 50 and the transmission image range 55. FIG. It is a flowchart of the image processing which PC1 performs. It is a flowchart of the intention determination process performed during image processing. It is explanatory drawing for demonstrating a process when the speaker is located in the center area | region 65 of the transmission image range 55 after the movement of an imaging range. It is a flowchart of the accident response process performed during image processing. It is explanatory drawing for demonstrating the process which restores the transmission image range 55 to the imaging range before a movement. It is explanatory drawing for demonstrating the process when the transmission image range 55 cannot be decompress | restored.

  A personal computer (hereinafter referred to as “PC”) 1 that is an embodiment of an image processing apparatus according to the present invention will be described below with reference to the drawings. The drawings to be referred to are used for explaining technical features that can be adopted by the present invention. The configuration of the apparatus, the flowcharts of various processes, and the like described in the drawings are not intended to be limited to these, but are merely illustrative examples.

  A schematic configuration of the PC 1 and the video conference terminal 3 will be described with reference to FIG. The PC 1 is a general PC that performs various information processing such as data communication and image display. 1 is a notebook PC and includes a display device 26 and an operation unit 27. Needless to say, a desktop PC that does not include devices such as a display device and an operation unit may be used. The PC 1 is connected to the video conference terminal 3 by, for example, a USB cable or infrared communication.

  The video conference terminal 3 includes a rotation shaft at the upper end. The user can switch the posture of the housing between the posture at the time of use and the posture at the time of non-use by rotating a part of the housing around the rotation axis. The video conference terminal 3 includes a microphone 45, a speaker 46, a camera 47, and an operation unit 48. When the user switches the posture of the housing to the posture at the time of use, all of the microphone 45, the speaker 46, the camera 47, and the operation unit 48 face the front of the video conference terminal 3. The video conference terminal 3 inputs the voice of the installed base from the microphone 45 and inputs the image from the camera 47. The input voice and image data is transmitted to the PC 1. The video conference terminal 3 generates sound from the speaker 46 based on the sound data received from the PC 1. The operation unit 48 is provided with a power button, a volume control button, a microphone mute button, and the like.

  The user can execute a remote conference (video conference) using an image by using the PC 1 and the video conference terminal 3. Specifically, the PC 1 transmits the voice and image data of its own site input from the video conference terminal 3 to a communication device such as a PC arranged at another site via a network 8 (see FIG. 2) such as the Internet. Send. At the same time, the PC 1 receives the voice and image data of the other base from the communication device arranged at the other base. The PC 1 causes the display device 26 to display the image of another base based on the received image data. Further, the PC 1 causes the speaker 46 of the connected video conference terminal 3 to generate the voice of the other base based on the received voice data. As a result, voices and images of a plurality of bases are shared, and the conference proceeds smoothly even when all users are not at the same base.

  The configurations of the PC 1 and the video conference terminal 3 can be changed. For example, a video conference may be executed without using the video conference terminal 3 by connecting a small camera to a PC including a microphone, a speaker, and a display device. A PC with a built-in camera may be used. Instead of the PC 1, an apparatus dedicated to video conferencing having a communication function and an image processing function may be used. In order to execute a video conference, the number of PCs 1 constituting the communication system may be two or more. Further, the present invention can be applied to both a P2P (peer to peer) communication system and a server communication system. In a P2P communication system, data is directly transmitted and received between a plurality of PCs 1. In a server-type communication system, data is transmitted and received via a server (not shown) that controls communication within the system.

  With reference to FIG. 2, the electrical configuration of the PC 1 and the video conference terminal 3 will be described. The PC 1 includes a CPU 10 that controls the PC 1. A ROM 11, a RAM 12, a hard disk drive (hereinafter referred to as “HDD”) 13, and an input / output interface 14 are connected to the CPU 10 via a bus 19.

  The ROM 11 stores a program for operating the PC 1, initial values, and the like. The RAM 12 temporarily stores various information used in the control program. The HDD 13 is a non-volatile storage device. The HDD 13 stores an image processing program for executing image processing to be described later. Instead of the HDD 13, a storage device such as an EEPROM or a memory card may be used. A video output processing unit 21, an external communication I / F 22, a USB I / F 23, and an operation unit 27 are connected to the input / output interface 14. The video output processing unit 21 processes the operation of the display device 26 that displays video. The external communication I / F 22 connects the PC 1 to the network 8. The USB I / F 23 connects the PC 1 to the video conference terminal 3 via a USB cable.

  The video conference terminal 3 includes a CPU 30 that controls the video conference terminal 3. A ROM 31, a RAM 32, a flash memory 33, and an input / output interface 34 are connected to the CPU 30 via a bus 39.

  The ROM 31 stores a program for operating the video conference terminal 3, an initial value, and the like. The RAM 32 temporarily stores various information. The flash memory 33 is a nonvolatile storage device. Connected to the input / output interface 34 are a USB I / F 41, an audio input processing unit 42, an audio output processing unit 43, a video input processing unit 44, and an operation unit 48. The USB I / F 41 connects the video conference terminal 3 to the PC 1. The voice input processing unit 42 processes input of voice data from the microphone 45. The audio output processing unit 43 processes the operation of the speaker 46. The video input processing unit 44 processes input of image data from the camera 47.

  The acquired image range 50 and the transmission image range 55 will be described with reference to FIG. The acquired image range 50 is an imaging range of image data acquired by the PC 1 from the video conference terminal 3 at its own base. That is, the captured image range of the image captured by the camera 47 of the video conference terminal 3 is the acquired image range 50. The transmission image range 55 is an imaging range of image data that the PC 1 transmits to a communication device at another base. If the PC 1 is within the range of the acquired image range 50, the transmission image range 55 can be freely set. Therefore, the PC 1 can change the range of the image displayed on the display device at the other base without changing the position and orientation of the video conference terminal 3, and can also enlarge and reduce the image.

  In the example illustrated in FIG. 3, a whiteboard 61, a participant 62, a participant 63, and the like are included in the acquired image range 50. The transmission image range 55 is set to a certain area including the center in the acquired image range 50, and the whiteboard 61, the participant 62, and the participant 63 are also included in the acquired image range 50. The PC 1 transmits image data for displaying the transmission image range 55 at another site to the communication device at the other site. As a result, the transmission image range 55 is displayed on the display device at the other site.

  In the present embodiment, a reference point 56 is set for determining whether or not the imaging range (that is, the acquired image range 50) of the camera 47 has moved. The PC 1 only needs to set the reference point 56 for a specific imaged object included in the background image within the acquired image range 50, and the imaged object for setting the reference point 56 may be one or more. For example, the PC 1 acquires a background image by capturing in advance only a background that does not include a participant before the start of the video conference. The PC 1 accurately determines whether or not the imaging range of the camera 47 has moved by setting the reference point 56 to the imaged object in the background image and monitoring the position of the reference point 56 in the acquired image range 50. be able to. Further, the PC 1 sets a plurality of reference points in preparation for the case where the participant moves and the reference point 56 is hidden, and when the camera 47 moves and the reference point 56 is out of the acquired image range 50. Can be set. By setting a plurality of reference points 56, it is possible to determine whether or not the acquired image range 50 has moved even when some of the reference points 56 are hidden. Further, the PC 1 can reset the reference point 56. Therefore, even when the participant and the camera 47 move, the movement amount of the acquired image range 50 can be determined more accurately. In the example illustrated in FIG. 3, the PC 1 performs image processing on the image data acquired from the video conference terminal 3 and specifies and identifies the corner of the rectangular imaging object among the imaging objects within the acquired image range 50. The angle is set to the reference point 56. As a result, in the example shown in FIG. 3, reference points 56 are set at the four corners of the whiteboard 61. Although details will be described later, the PC 1 acquires the position of the reference point 56 in the acquired image range 50 as movement information indicating the movement state of the imaging range of the camera 47. Based on the position of the acquired reference point 56, it is determined whether or not the acquired image range 50, which is the imaging range of the camera 47, has been moved. If it has moved, the amount of movement is calculated. The PC 1 may set the reference point 56 according to the operation of the operation unit 27 by the user.

  With reference to FIGS. 4 to 9, image processing executed by the PC 1 according to the present embodiment will be described. An image processing program is installed in the HDD 13 of the PC 1 as an application for executing a video conference. When the CPU 10 of the PC 1 inputs an instruction to start an application from the operation unit 27 and starts a video conference with a communication device at another base, the CPU 10 executes the image processing shown in FIG.

  As illustrated in FIG. 4, the CPU 10 that has started image processing acquires image data of an image captured by the camera 47 from the video conference terminal 3. A feature point is specified by performing image processing on the acquired image data. The CPU 10 sets the identified feature point as the reference point 56 (S1). As described above, the PC 1 according to the present embodiment can determine the movement of the imaging range by setting a plurality of reference points 56 even when some reference points 56 are hidden. Next, the CPU 10 acquires image data of the latest image captured by the camera 47 from the video conference terminal 3 (S2). The CPU 10 calculates the coordinates of the reference point 56 in the acquired image range 50 (S3).

  If no reference point 56 has been detected (S4: YES), the CPU 10 determines that the acquired image range 50 that is the imaging range of the camera 47 has moved (S5). Next, as in the process of S1, a plurality of reference points 56 are reset (S6). The PC 1 stops the transmission of the captured image data to the communication device at the other base in order to prevent the disturbed image being moved from being displayed at the other base (S7). The PC 1 adds a movement time that is the time during which the acquired image range 50 is moving (S8), and the process proceeds to the determination of S9. If the instruction to end the video conference is not input to the operation unit 27 (S9: NO), the process returns to S2.

  When the CPU 10 can detect at least one reference point 56 (S4: NO), each coordinate of the latest reference point 56 calculated in S3 and the reference point 56 calculated in the previous processing of S3. The average value of the distances from the respective coordinates is calculated as the amount of change in the position of the reference point 56 and added (S10). The CPU 10 determines whether or not the acquired image range 50 has moved based on the amount of change calculated in S10 (S11). When the amount of change calculated in S10 is equal to or greater than a certain threshold, the CPU 10 determines that the acquired image range 50 has moved (S11: YES), stops transmitting image data (S7), and adds the moving time. (S8). The process proceeds to determination of S9. If the instruction to end the video conference is not input to the operation unit 27 (S9: NO), the process returns to S2, and the movement amount and the movement time are continuously added until the movement of the acquired image range 50 is completed (S8 and S10).

  If the amount of change calculated in S10 is less than the threshold, the CPU 10 determines that the acquired image range 50 has not moved (S11: NO), and determines whether the acquired image range 50 has moved until just before. (S12). When it is determined that the acquired image range 50 has moved in the previous determination of S4 / S5 or S11 (S12: YES), the CPU 10 resets the reference point 56 (S13), and performs intentional determination processing. Perform (S14). In the intentional determination process, it is determined whether or not the user has intentionally moved the imaging range of the camera 47.

  As shown in FIG. 5, the CPU 10 that has started the intentional determination process determines whether or not the video conference terminal 3 has been operated within the travel time (S31). In the present embodiment, the CPU 10 has a USB cable for connecting or disconnecting the video conference terminal 3 and the PC 1 when any of the operation units 48 of the video conference terminal 3 is operated within the movement time. In this case, it is determined that the video conference terminal 3 has been operated. However, for example, when the video conference terminal 3 can be equipped with a memory card, the attachment / detachment of the memory card may be determined as an operation of the video conference terminal 3. The user rarely intentionally simultaneously operates the video conference terminal 3 and moves the imaging range. Therefore, when it is determined that the video conference terminal 3 is operated within the moving time (S31: YES), the CPU 10 determines that the cause of the movement of the acquired image range 50 is an accident (S38), and the process returns to the image processing.

  When the video conference terminal 3 is not operated within the predetermined time (S31: NO), the CPU 10 calculates the amount of vertical movement of the acquired image range 50 from the amount of change in the position of the reference point 56 added. It is determined whether or not the calculated vertical movement amount is equal to or greater than a threshold value A (S32). When the user intentionally moves with the video conference terminal 3, the video conference terminal 3 is accompanied by movement in the vertical direction. Therefore, if the amount of movement in the vertical direction is greater than or equal to the threshold A (S32: YES), the CPU 10 determines that the user has intentionally moved the imaging range of the camera 47 (S39). Processing returns to image processing.

  If the amount of movement of the acquired image range 50 in the vertical direction is less than the threshold A (S32: NO), the CPU 10 calculates the amount of movement of the acquired image range 50 in the horizontal direction from the amount of change in the position of the reference point 56. It is determined whether or not the calculated horizontal movement amount is less than the threshold value B (S33). If the amount of movement in the horizontal direction is less than the threshold B (S33: YES), the CPU 10 determines whether or not the added movement time is less than the threshold C (S34). When finely adjusting the imaging range of the camera 47, the user takes a long time and carefully adjusts the position and direction of the camera 47. When the movement amount of the acquired image range 50 is small and the movement time is short, there are many cases where the user or the like suddenly touches the video conference terminal 3 to move the imaging range. Therefore, when the amount of movement in the horizontal direction is equal to or greater than the threshold value B (S33: NO), or when the movement time is equal to or greater than the threshold value C (S34: NO), the user deliberately captures the camera 47. It is determined that the range has been moved (S39). Processing returns to image processing. In some cases, the video conference terminal 3 moves greatly due to an accident (for example, the video conference terminal 3 falls down). In this case, the user is likely to notice the movement due to an impact sound or the like, and thus the image is distorted. It is unlikely that a video conference will continue.

  If the horizontal movement amount is less than the threshold value B (S33: YES) and the movement time is less than the threshold value C (S34: YES), the CPU 10 performs image processing on the image data acquired from the video conference terminal 3. And the face area of the imaged participant is detected (S35). Various well-known methods can be used to detect the face area. For example, a method of detecting parts such as eyes, nose, and mouth by feature points may be used, or a method of comparing with base data stored in advance to determine a face may be used. Next, the CPU 10 detects the position of the speaker from the detection result of the face area and the voice input by the microphone 45 of the video conference terminal 3 (S36). Specifically, when the speech is input from the microphone 45, the CPU 10 detects the position of the participant whose mouth of the detected face area is moving as the position of the speaker. The CPU 10 determines whether or not the detected position of the speaker is within the central area in the transmission image range 55 (S37).

  FIG. 6 shows an example of the acquired image range 50 moved from the state shown in FIG. The center region 65 is a predetermined region including the center point of the transmission image range 55, and the shape and size of the center region 65 can be set as appropriate. In the example shown in FIG. 6, the acquired image range 50 has moved from the position shown in FIG. 3, but the participant 62 identified as the speaker is located in the central region 65. As a result, the participant 62 who is speaking is displayed at the center of the image displayed on the display device at the other site. In this case, the user may intentionally move the video conference terminal 3 so that the participant 62 who is speaking is positioned at the center of the image. Even when the video conference terminal 3 is not intentionally moved, if the participant 62 who is speaking is displayed at the center of the image, there is a low possibility that the progress of the video conference will be hindered.

  Accordingly, when the CPU 10 determines that the detected speaker position is within the central area 65 (S37: YES), the CPU 10 determines that the user has intentionally moved the imaging range of the camera 47 (S39). Although details of the process will be described later, if the CPU 10 determines that the movement of the imaging range is intentional, the process of restoring the position of the transmission image range 55 is not performed and the transmission of the image data is continued as it is. As a result, as shown in FIG. 6, the participant 62 who is speaking is located at the center of the transmission image range 55 that is the range of the image displayed on the display device at the other site. On the other hand, when the CPU 10 determines that the position of the speaker is not within the central area 65 (S37: NO), the CPU 10 determines that the cause of the movement of the acquired image range 50 is an accident (S38), and the process returns to the image processing.

  Returning to the description of FIG. When the intentional determination process (S14) ends, the CPU 10 determines whether or not the determination result in the intentional determination process is “intentional” (S15). If it is determined that the user has intentionally moved the imaging range (S15: YES), the CPU 10 resumes transmission of the captured image data to the communication device at the other base (S17), and the process proceeds to the determination of S9. To do. If it is determined that the cause of the movement of the imaging range is an accident (S15: NO), the CPU 10 executes an accident response process (S16), and the process proceeds to the determination of S9. In the accident handling process, a process for restoring the transmission image range 55, a process for changing the image data to be transmitted to other image data, and the like are performed.

  As shown in FIG. 7, when the accident response process is started, the CPU 10 searches the latest acquired image range 50 for all the reference points 56 detected before the movement (S41). When all the reference points 56 exist in the acquired image range 50 (S42: YES), the CPU 10 determines whether or not the transmission image range 55 can be restored to the range before the movement (S43).

  In FIG. 8 and FIG. 9, only the reference point 56 at the upper left corner of the whiteboard 61 is illustrated among the plurality of reference points 56 for the sake of simplicity. In the following description, only the reference point 56 shown in FIGS. 8 and 9 will be used. In the example shown in FIG. 8, the searched position of the reference point 56 after movement in the transmission image range 55 matches the position where the reference point before movement (reference point 58 before restoration) existed. Even if the set transmission image range (transmission image range 57 before restoration) is moved and a new transmission image range 55 is set, the new transmission image range 55 falls within the acquired image range 50. That is, in the example shown in FIG. 8, the reference point 56 exists at the upper left part of the transmission image range 55 before the movement. Therefore, in order to restore the transmission image range 55, the reference point is also used in the restored transmission image range 55. The position of 56 needs to be located in the upper left part of itself. Therefore, the CPU 10 must move the transmission image range 55 from the position of the transmission image range 57 before restoration to the lower right. In the example illustrated in FIG. 8, since the movement amount of the acquired image range 50 is small, the transmission image range 55 does not protrude from the acquired image range 50 even after being moved (restored). In this case, the CPU 10 determines that the transmission image range 55 can be restored.

  On the other hand, in the example shown in FIG. 9, it is assumed that the transmission image range 55 is moved so that the position of the reference point 56 in the transmission image range 55 becomes the position of the reference point 58 before restoration in the transmission image range 57 before restoration. To do. In this case, the right end portion of the transmission image range 55 is outside the acquired image range 50. Therefore, in the example illustrated in FIG. 9, the CPU 10 determines that the transmission image range 55 cannot be restored.

  Returning to the description of FIG. When the CPU 10 determines that the transmission image range 55 can be restored (S43: YES), the position of each reference point 56 in the transmission image range 55 matches the position where each reference point 56 existed before the movement. As described above, the position of the transmission image range 55 in the acquired image range 50 is moved (S44), and the transmission image range 55 is enlarged or reduced to match the sizes (S45). Processing returns to image processing. Thereby, the transmission image range 55 is restored. As can be seen by comparing the lower diagram in FIG. 3 with the lower diagram in FIG. 8, the display images at the other bases display the same image as before the imaging range of the camera 47 moves. The user does not have to return the position of the camera 47 to the original position.

  If the CPU 10 determines that the transmission image range 55 cannot be restored (S43: NO), the CPU 10 starts to transmit the data of the in-correction notification image 68 to the display devices at other bases (S47). The in-correction notification image 68 is an image for notifying users at other bases that the imaging range of the camera 47 has moved, as shown in the lower part of FIG. 9. In the present embodiment, an image displaying a message “Currently correcting camera field of view” is used as the in-correction notification image 68. It should be noted that users at other bases can grasp that the imaging range has moved, and do not see useless images unrelated to the conference. Further, when the imaging range is moved, there is a possibility that image data (for example, confidential material data) that is inconvenient when viewed by users at other bases may be displayed. However, the PC 1 can improve the security by transmitting the data of the in-correction notification image 68 instead of the image data of its own base.

  Further, the CPU 10 starts displaying the movement notification image on the display device 26 at its own base (S48). The movement notification image is an image for notifying the user at the local site that the imaging range of the camera 47 has moved. In the present embodiment, the message “terminal has moved” is displayed together with images of other bases. The CPU 10 starts measuring the time (notification time) during which the in-correction notification image 68 and the movement notification image are displayed (S49), and the processing returns to the image processing.

  Returning to the description of FIG. When the acquired image range 50 has not moved (S11: NO) and the acquired image range 50 has not moved immediately before (S12: NO), the CPU 10 adds the movement time added in S8, and The amount of change added in S10 is initialized to “0” (S19). The CPU 10 determines whether or not it is informing that the imaging range has moved (S20). When the notification time is not measured and it is determined that the notification is not being performed (S20: NO), the process returns to the determination of S9 as it is. When the notification time is measured and it is determined that the notification that the imaging range has moved is being performed (S20: YES), the CPU 10 determines whether or not the notification time has reached a certain time (S21). If the predetermined time has not been reached (S21: NO), the process proceeds to the determination of S9 as it is, and the notification is continued. When the notification time reaches a certain time (S21: YES), the CPU 10 ends the transmission of the data of the correction notification image 68 (S22). The CPU 10 ends the display of the movement notification image on the display device 26 at the local site (S23). Furthermore, data transmission of the captured image of the camera 47 is resumed (S24). Next, the notification time is initialized (S25), and the process proceeds to the determination of S9. When an instruction to end the video conference is input from the operation unit 27 or the communication device at another base (S9: YES), the image processing ends.

  As described above, according to the PC 1 according to the present embodiment, when it is determined that the movement of the imaging range of the camera 47 is due to an accident and the transmission image range 55 can be restored, the transmission image range 55 is It is restored to the imaging range before movement. Therefore, the user does not have to return the imaging range of the camera 47 himself. If the transmission image range 55 cannot be restored, the PC 1 can stop the data transmission of the captured image (the frame rate is set to “0”) and can transmit the data of the in-correction notification image 68 instead. As a result, it is possible to notify the user that the imaging range has moved unintentionally, and it is possible to prevent inconvenience due to transmission of image data that does not need to be transmitted. When the PC 1 determines that the user has intentionally moved the imaging range, processing such as restoration of the transmission image range 55 and change of image data is not performed. Therefore, the user can move the transmission image range 55 as intended. Thus, the PC 1 can perform appropriate processing according to the situation when the imaging range of the camera 47 changes. In particular, in the case of using a small imaging device that is placed on a plane and used like the video conference terminal 3, the imaging range is easily moved unexpectedly. However, according to the PC 1 of the present embodiment, a smooth video conference can be easily realized using a small imaging device.

  When the user intentionally moves with the video conference terminal 3, the video conference terminal 3 is accompanied by movement in the vertical direction. Therefore, the PC 1 can determine whether or not the movement of the imaging range is intentional based on the amount of movement of the imaging range in the vertical direction. In addition, when the moving amount of the imaging range is small and the moving time is short, the imaging range often moves when the user or the like suddenly touches the video conference terminal 3. Therefore, the PC 1 can appropriately determine whether or not the movement of the imaging range is intentional based on the movement amount and the movement time of the imaging range.

  The PC 1 moves the transmission image range 55 so that the position of the reference point 56 in the transmission image range 55 is the same before and after the movement of the imaging range. As a result, the transmission image range 55 can be restored easily and reliably. Further, when the imaging range of the camera 47 is moved, if the position of the speaker is within the central area 65 of the transmission image range 55, users at other bases can visually recognize the speaker. In this case, the PC 1 can smoothly advance the remote conference by determining that the user has intentionally moved the imaging range.

  The user rarely intentionally operates and moves the video conference terminal 3 simultaneously. Therefore, the PC 1 can make an accurate determination by determining that the movement when the video conference terminal 3 is operated is an accident. The PC 1 determines whether or not the imaging range of the camera 47 has moved based on the image data acquired from the video conference terminal 3. Therefore, even when the video conference terminal 3 does not include an acceleration sensor or the like, the PC 1 can make an accurate determination using image data used for the video conference.

  In the above embodiment, the PC 1 corresponds to the “image processing apparatus” of the present invention. The camera 47 of the video conference terminal 3 corresponds to the “imaging means” of the present invention. The CPU 10 that acquires image data in S2 of FIG. 4 functions as an “image data acquisition unit”. The CPU 10 that acquires the change amount and the movement time in S3, S8, and S10 of FIG. 4 functions as a “movement information acquisition unit”. The CPU 10 that determines whether or not the imaging range has moved in S4, S5, and S11 in FIG. 4 functions as a “movement determination unit”. The CPU 10 that performs the intentional determination process shown in FIG. 5 functions as “intentional determination means”. The CPU 10 that determines whether or not the transmission image range 55 can be restored in S43 of FIG. 7 functions as a “restoration judgment unit”. The CPU 10 that restores the transmission image range 55 in S44 and S45 of FIG. 7 functions as a “restoring means”. The CPU 10 that transmits the data of the in-correction notification image 68 in S47 of FIG. 7 functions as “processing means”.

  The threshold A corresponds to the “first threshold”, the threshold B corresponds to the “second threshold”, and the threshold C corresponds to the “third threshold”. The CPU 10 that specifies the position of the reference point 56 in S10 of FIG. 4 and S41 of FIG. 7 functions as “specifying means”. An imaged object in which the reference point 56 is set (the whiteboard 61 in the examples of FIGS. 3, 8, and 9) corresponds to the “reference imaged object” of the present invention. The CPU 10 that detects the position of the speaker in S36 of FIG. 5 functions as a “detection unit”. The center area 65 corresponds to a “predetermined area”.

  The process of acquiring image data in S2 of FIG. 4 corresponds to an “image data acquisition step”. The process of acquiring the change amount and the movement time in S3, S8, and S10 of FIG. 4 corresponds to a “movement information acquisition step”. The process of determining whether or not the imaging range has moved in S4, 5 and 11 in FIG. 4 corresponds to a “movement determination step”. The intentional determination process shown in FIG. 5 corresponds to a “intentional determination step”. The process of determining whether or not the transmission image range 55 can be restored in S43 of FIG. The process of restoring the transmission image range 55 in S44 and S45 of FIG. 7 corresponds to a “restoration step”. The process of transmitting the data of the in-correction notification image 68 in S47 of FIG. 7 corresponds to a “processing step”.

  It goes without saying that the present invention is not limited to the above-described embodiment, and various modifications are possible. For example, the PC 1 of the above embodiment restores the imaging range of the image data to be transmitted to the communication device at another base by changing the position and size of the transmission image range 55 within the acquired image range 50. . However, the method for restoring the transmission image range 55 is not limited to this. For example, when the video conference terminal 3 can change the orientation of the camera 47 (pan / tilt) with a motor or the like, the PC 1 transmits an instruction to change the orientation of the camera 47 to the video conference terminal 3, thereby transmitting images. The range 55 may be restored. In this case, when determining whether or not the transmission image range 55 can be restored in S43 of FIG. 7, the PC 1 searches the surroundings of the reference point 56 while changing the orientation of the camera 47. Also good. As a result, the PC 1 can restore the transmission image range 55 in a wider range.

  When the transmission image range 55 cannot be restored, the PC 1 of the above embodiment stops transmission of the captured image data (the frame rate is set to “0”), and instead uses the data of the in-correction notification image 68 at the other bases. Send to communication device. However, this process can be changed. For example, the PC 1 may only stop transmission of captured image data. By simply stopping the transmission of the captured image data, it is possible to eliminate the possibility that confidential information will be visually recognized by users at other locations and the possibility that unnecessary image data will be transmitted to the conference. Further, the PC 1 may prevent leakage of confidential information by lowering the resolution of captured image data to be transmitted. The PC 1 may reduce the discomfort of the user due to visually recognizing unnecessary images by reducing the frame rate.

  The contents of the intentional determination process shown in FIG. 5 can also be changed. For example, the present invention can be realized even if the processing order of S31 to S34 and S37 is changed. Moreover, you may abbreviate | omit any process of S31-S34, S37. For example, the present invention can be realized even if the processing of S31 and the processing of S35 to S37 are omitted. Needless to say, the thresholds A to C can be set as appropriate. The user may be able to change the thresholds A to C.

  The PC 1 of the above embodiment determines whether the imaging range of the camera 47 has moved using the image data acquired from the video conference terminal 3. However, the movement of the imaging range may be determined using other methods. For example, if a camera provided with an acceleration sensor is used, the PC 1 can determine whether the camera has moved based on the output value of the acceleration sensor. The PC 1 may output a sound for notifying that the imaging range has moved, instead of or together with the correction notification image and the movement notification image. Further, the PC 1 may determine the movement of the imaging range using the color distribution of the image. For example, the PC 1 calculates the color distribution from the acquired image data after acquiring the image data in S2 of FIG. The color distribution calculated this time is compared with the color distribution calculated last time, and when the amount of change in the color distribution is equal to or greater than the threshold, it is determined that the imaging range has moved. When it is determined that the imaging range has moved, the movement amount of the imaging range may be calculated by comparing the positions of the reference points 56 before and after the movement. Further, the PC 1 of the above-described embodiment acquires a background image by capturing an image of the background in advance, and sets the reference point 56 on the imaged object in the background image. However, the PC 1 may specify a background portion from images captured by a plurality of cameras without capturing a background image in advance. As described above, the method for determining the movement of the imaging range and the method for calculating the movement amount can be changed as appropriate.

1 PC
3 Video conference terminal 26 Display device 41 USB I / F
47 Camera 48 Operation unit 50 Acquired image range 55 Transmitted image range 56 Reference point 65 Central area 68 Notification image being corrected

Claims (9)

An image processing apparatus that processes image data to be transmitted to a communication apparatus arranged at another base in order to perform a remote conference by users at a plurality of bases,
Image data acquisition means for acquiring image data of an image captured by an imaging means for capturing an image;
Movement information acquisition means for acquiring movement information indicating the movement state of the imaging range of the imaging means;
Movement determining means for determining whether or not the imaging range of the imaging means has moved according to the movement information acquired by the movement information acquiring means;
When the movement determination unit determines that the imaging range has moved, using at least one of the movement information acquired by the movement information acquisition unit and operation state information indicating an operation state for the imaging unit, Deliberate judgment means for judging whether or not the user has intentionally moved the imaging range;
Whether or not the transmission image range, which is the imaging range of the image data to be transmitted to the communication device, can be restored to the imaging range before the movement when it is determined that the intentional determination means does not intentionally move the image, Restoration determination means for determining using image data acquired by the data acquisition means;
When it is determined that the transmission image range can be restored by the restoration judgment unit, the restoration unit restores the transmission image range to the imaging range before movement;
At least one of processing for changing an image displayed on a display device and processing for changing a frame rate of image data transmitted to the communication device when the restoration determination means determines that the transmission image range cannot be restored. An image processing apparatus comprising: processing means for performing any of the above.   The deliberate determination means determines that the user has intentionally moved the imaging range when the amount of movement in the vertical direction indicated by the movement information acquired by the movement information acquisition means is greater than or equal to a first threshold value. The image processing apparatus according to claim 1.   The deliberate determination means has a vertical movement amount indicated by the movement information acquired by the movement information acquisition means that is less than the first threshold value, a horizontal movement amount that is less than a second threshold value, and the movement The image processing apparatus according to claim 2, wherein when the moving time of the imaging range indicated by the information is less than a third threshold, the user determines that the imaging range is not intentionally moved. The restoration means includes
Of the image data acquired by the image data acquisition means, comprising a specifying means for specifying the same reference imaging object in each of the image data before and after it is determined that the imaging range has been moved by the movement determination means,
The position in the imaging range of the reference imaged object specified by the specifying unit in the image data after movement matches the position of the reference imaged object specified by the specifying unit in the image data before movement. The image processing apparatus according to claim 1, wherein the transmission image range is restored by moving. The deliberate judgment means is
A detecting means for detecting a position of a speaker in the transmission image range;
If the position of the speaker detected by the detection unit after the movement is detected by the movement determination unit is within a predetermined region including the center point of the transmission image range, the user intentionally moves the imaging range. The image processing apparatus according to claim 1, wherein the image processing apparatus determines that the image processing has been performed.   6. The intention determination unit according to claim 1, wherein when the operation state information indicates that the imaging unit has been operated, it is determined that the user has not intentionally moved the imaging range. An image processing apparatus according to 1.   The processing means changes the image data to be transmitted to the communication device from image data acquired by the image data acquisition means to image data including an image for notifying that the imaging range of the imaging means has moved. The image processing apparatus according to claim 1, wherein: An image processing method performed by an image processing apparatus that processes image data to be transmitted to a communication apparatus arranged at another base in order to execute a remote conference by users at a plurality of bases,
An image data acquisition step of acquiring image data of an image captured by an imaging means for capturing an image;
A movement information acquisition step of acquiring movement information indicating a movement state of the imaging range of the imaging means;
A movement determination step of determining whether or not the imaging range of the imaging means has moved according to the movement information acquired in the movement information acquisition step;
When it is determined that the imaging range has moved in the movement determination step, using at least one of the movement information acquired in the movement information acquisition step and operation state information indicating an operation state for the imaging unit, A deliberate determination step for determining whether or not the user has intentionally moved the imaging range;
Whether it is possible to restore the transmission image range, which is the imaging range of the image data to be transmitted to the communication device, to the imaging range before the movement when it is determined that it is not intentionally moved in the intention determination step, A restoration determination step for determining using the image data acquired in the data acquisition step;
When it is determined that the transmission image range can be restored in the restoration determination step, a restoration step of restoring the transmission image range to an imaging range before movement;
At least one of processing for changing an image displayed on a display device and processing for changing a frame rate of image data transmitted to the communication device when it is determined in the restoration determination step that the transmission image range cannot be restored. An image processing method comprising: a processing step for performing any one of the steps. An image processing program used in an image processing apparatus for processing image data to be transmitted to a communication device arranged in another base in order to execute a remote conference by users at a plurality of bases,
An image data acquisition step of acquiring image data of an image captured by an imaging means for capturing an image;
A movement information acquisition step of acquiring movement information indicating a movement state of the imaging range of the imaging means;
A movement determination step of determining whether or not the imaging range of the imaging means has moved according to the movement information acquired in the movement information acquisition step;
When it is determined that the imaging range has moved in the movement determination step, using at least one of the movement information acquired in the movement information acquisition step and operation state information indicating an operation state for the imaging unit, A deliberate determination step for determining whether or not the user has intentionally moved the imaging range;
Whether it is possible to restore the transmission image range, which is the imaging range of the image data to be transmitted to the communication device, to the imaging range before the movement when it is determined that it is not intentionally moved in the intention determination step, A restoration determination step for determining using the image data acquired in the data acquisition step;
When it is determined that the transmission image range can be restored in the restoration determination step, a restoration step of restoring the transmission image range to an imaging range before movement;
At least one of processing for changing an image displayed on a display device and processing for changing a frame rate of image data transmitted to the communication device when it is determined in the restoration determination step that the transmission image range cannot be restored. An image processing program including an instruction for causing a controller of the image processing apparatus to execute a processing step for performing any of the above.

Images