JP2021036440A - Information processing apparatus - Google Patents

Abstract

To automatically perform starting, setting, and the like of an application that cooperates with an imaging apparatus, on the basis of information obtained from the imaging apparatus.SOLUTION: In an information processing apparatus, an acquisition section acquires at least one of first information which is information on an image captured by an imaging section and second information which is information on an imaging range to be captured by the imaging section. An application control section provides a user with an application corresponding to the information acquired by the acquisition section, out of a plurality of applications which use the information on the image captured by the imaging section. On the basis of the image obtained from the imaging section or information on the imaging range, an application to be executed in cooperation with the imaging section is automatically started or settings required for executing the application are automatically performed.SELECTED DRAWING: Figure 5

Description

The present invention relates to a technique for automatically starting and setting an application in a terminal device connected to a photographing device.

There are various applications that connect a camera to a PC and execute it using the captured image. For example, videophone is one example. Further, Patent Document 1 discloses an application that enables an operation (click, selection, execution, etc.) according to the detected hand movement by detecting the hand movement of the PC user.

Special Table 2010-500465

There may be a plurality of applications that can operate in cooperation with one camera. In this case, in order to switch from one application to another, the user needs to manually launch the application or make settings for that application after changing the camera installation position and so on. There was a problem that it was troublesome.

As an example of the problem to be solved by the present invention, the above is given as an example. An object of the present invention is to automatically start and set an application that operates in cooperation with the photographing device based on the information obtained from the photographing device.

The invention according to the claim is an information processing apparatus, which is at least one of first information which is information of an image taken by a photographing unit and second information which is information about a photographing range photographed by the photographing unit. The acquisition unit is provided with an application control unit that provides the user with an application corresponding to the information acquired by the acquisition unit among a plurality of applications that use the information of the image captured by the acquisition unit. It is characterized by.

The configuration of the interaction system according to the embodiment of the present invention is shown. It is a perspective view which shows the appearance of the photographing apparatus. The state of rotation by the moving part is shown. It is a block showing the functional configuration of a PC. It is a flowchart of an automatic setting process. An example of the installation state of the photographing device is shown. An example of the installation state of the photographing device is shown. It is a flowchart of application estimation processing.

In a preferred embodiment of the present invention, the information processing apparatus receives at least one of the first information which is the information of the image taken by the photographing unit and the second information which is the information about the photographing range photographed by the photographing unit. It includes an acquisition unit to be acquired and an application control unit that provides a user with an application corresponding to the information acquired by the acquisition unit among a plurality of applications that use the information of the image captured by the acquisition unit.

The above information processing device acquires at least one of the first information which is the information of the image taken by the photographing unit such as a camera and the second information about the photographing range, and provides the user with an application corresponding to the acquired information. provide. Therefore, it is possible to automatically provide an application executed in cooperation with the photographing unit based on the information regarding the image or the photographing range obtained from the photographing unit. Here, providing an application includes at least one of launching the application and making necessary settings for the application.

In one aspect of the information processing apparatus, the application control unit provides the user with a first application when the image captured by the imaging unit includes an information output device, and the imaging unit provides the first application. If the captured image includes an information input device, a second application is provided to the user. In this aspect, different applications are provided depending on whether the captured image includes an information output device or an information input device.

In another aspect of the information processing apparatus, the second information includes information about the shooting direction of the photographing unit, and the application control unit photographs an upper side with respect to a position where the photographing unit is fixed. If so, the third application is provided to the user, and if the fixed portion is not photographed above the fixed position, the fourth application is provided to the user. In this aspect, different applications are provided depending on whether the photographing unit is photographing above or below.

In another aspect of the information processing apparatus described above, the application control unit executes at least one of starting an application corresponding to the information acquired by the acquisition unit and setting for the application. In still another aspect, the application control unit makes settings for the photographing unit when executing an application corresponding to the information acquired by the acquisition unit. As a result, the application is started and necessary settings are automatically executed based on the information of the captured image or the information regarding the capture range.

In a preferred example of the information processing apparatus, the photographing unit includes an infrared light emitting function and an optical filter that transmits infrared rays and cuts visible light, and the application control unit turns on the infrared light emitting function. And off, and on and off of the optical filter.

In another preferred example, the information processing device includes an information input device or an information output device, and the second information is a relative positional relationship between the photographing unit and the information input device or the information output device. The acquisition unit generates information indicating the relative positional relationship by image analysis of the first information, including the information to be shown.

In another preferred embodiment of the present invention, the information processing method executed by the information processing apparatus connected to the photographing unit is the first information which is the information of the image photographed by the photographing unit, and the photographing unit. Corresponds to the acquisition process of acquiring at least one of the second information which is the information about the shooting range to be photographed by, and the information acquired by the acquisition unit among a plurality of applications using the information of the image photographed by the photographing unit. It has an application control process for providing an information processing application to a user. Also by this method, it is possible to automatically provide an application executed in cooperation with the photographing unit based on the information regarding the image or the photographing range obtained from the photographing unit.

In another preferred embodiment of the present invention, the information processing program connected to the photographing unit and executed by the information processing apparatus including the computer is the first information which is the information of the image photographed by the photographing unit, and the above-mentioned first information. An acquisition means for acquiring at least one of the second information which is information about a shooting range shot by the shooting unit, and information acquired by the acquisition unit among a plurality of applications using the information of the image shot by the shooting unit. The computer functions as an application control means for providing a corresponding application to a user. The above information processing apparatus can be realized by executing this program on a computer. This program can be stored in a storage medium.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.
[Constitution]
FIG. 1 shows the configuration of an interaction system according to an embodiment of the present invention. The interaction system includes a photographing device 10 and a PC 20 as an information processing device. The photographing device 10 and the PC 20 are connected by wire with a signal cable or the like, or are connected by wireless communication. In the interaction system, the captured image captured by the photographing device 10 is transmitted to the PC 20. A plurality of applications that operate in cooperation with the photographing device 10 are installed on the PC 20. When the user installs the photographing device 10 at a predetermined position and the photographed image is transmitted from the photographing device 10 to the PC 20, the PC 20 receives the photographed image from a plurality of applications installed inside the user. Estimates the application that it wants to run, and automatically launches the application and makes the necessary settings to run the application.

FIG. 2 shows the appearance of the photographing apparatus 10. The photographing device 10 includes a main body portion 11, a movable portion 12, and a fixed portion 14. The fixing portion 14 is a portion for fixing the photographing device 10 in some form. In this embodiment, the fixing portion 14 is configured as a clip so that it can be fixed by being sandwiched between the screen edges of the PC 20 or placed on a desk as a stand.

The movable portion 12 is a portion for the user to adjust the direction and position of the main body portion 11, and includes a rotating portion 12a, a supporting portion 12b, and a rotating shaft 12c. The rotating portion 12a is rotatable around the rotating shaft 12c with respect to the supporting portion 12b. The rotating portion 12a is fixed to the main body portion 11, and the supporting portion 13 is fixed to the fixing portion 14. As a result, the main body portion 11 can rotate with respect to the fixed portion 14. In this embodiment, the rotating unit 12 is configured to be capable of rotating only one axis, but may be provided with a multi-axis rotating mechanism or a sliding mechanism.

The main body 11 includes a plurality of light emitting units 15 and a pair of camera modules 16. The main body 11 has various shooting functions depending on the application executed on the PC 20 side. In this embodiment, the main body 11 is provided with a pair of camera modules 16 to form a stereo camera, and further includes three infrared LED light emitting units 15. Further, the camera module 16 is provided with an infrared transmission / visible light cut filter (hereinafter, simply referred to as “optical filter”) which can be switched on / off and is not shown. The on / off of the optical filter is switched by the control signal supplied from the PC 20.

FIG. 3 shows a state of rotation by the movable portion 12. FIG. 3A shows a state in which the main body 11 is erected substantially perpendicular to the fixed portion 14, and the camera module 16 of the main body 11 photographs the direction of the arrow. FIG. 3B shows a state in which the main body 11 is rotated 180 degrees around the rotation axis 12c with respect to the fixed portion 14, and the camera module 16 of the main body 11 corresponds to the rotation state (rotation angle). Take a picture of the direction of the arrow. The movable portion 12 is provided with a rotation angle detecting mechanism such as a rotary encoder (not shown), and detects the rotation angle of the rotating portion 12a in a state set by the user.

FIG. 4 is a block diagram showing a functional configuration of the PC 20. The PC 20 includes a communication unit 21, an input unit 22, a display unit 23, and a control unit 24. The communication unit 21 transmits and receives control signals and captured images to and from the photographing device 10 by wire or wireless communication. The input unit 22 is an information input device such as a keyboard and a mouse, and in this embodiment, it is a keyboard provided in the main body of the laptop PC and a mouse (not shown). The display unit 23 is an information output device such as a liquid crystal display, and in this embodiment, it is a liquid crystal display provided on the main body of the laptop PC.

The control unit 24 controls the entire configuration of the PC 20 in an integrated manner, and specifically, is composed of a CPU of the PC 20 and the like. The control unit 24 executes the automatic setting software and camera applications (hereinafter, also simply referred to as “applications”) A1 to A4 that operate in cooperation with the camera.

The automatic setting software estimates an application corresponding to the interaction method that the user wants to use based on the captured image by the photographing device 10, the state of the movable part 12 of the photographing device 10, and automatically starts the application or the application. It automatically sets the settings required for the application.

The camera applications A1 to A4 are software that realizes the interaction method provided by the interaction system, and are as follows in this embodiment, respectively.
(A1): Gesture operation application for the screen (A2): Gesture operation application near the keyboard (A3): Line-of-sight input application (A4): Web camera application

Specifically, the application A1 is an application in which the user performs an operation such as a menu or an icon displayed on the screen of the display unit 23 by performing a gesture such as pointing at the screen. The gesture by the user in this case does not need to touch the screen with the finger, and may be performed in the air. This application enables a touch panel-like operation. In addition, since the position and locus of the finger can be acquired from the captured image without the finger touching the screen, it is possible to realize an interaction that is impossible with the touch panel.

The application A2 is an application in which the user performs an operation by making a gesture such as creating a specific hand shape near the keyboard or lifting a finger from the keyboard and shaking it. With this application, you don't have to take your hands off the keyboard, so you can perform smooth operations while entering keys.

The application A3 is an application that operates the pointer according to the line of sight of the user. Specifically, the user's eye portion is extracted from the captured image of the user's face to detect the direction of the line of sight, and the operation input corresponding to the detected direction of the line of sight is enabled.

Application A4 is a so-called videophone-like application that interacts with a remote location with a moving image. When no dialogue is performed, the image captured by the photographing device 10 is only displayed on the display unit 23.

In addition, each application can receive the setting for itself from the automatic setting software.

[Automatic setting process]
Next, the automatic setting process by the automatic setting software will be described. FIG. 5 is a flowchart of the automatic setting process. The automatic setting process is repeatedly executed by the control unit 24 such as the CPU of the PC 20 executing the automatic setting software.

First, the control unit 24 acquires an image captured by the photographing device 10 (step S11). Specifically, the control unit 24 receives the captured image from the photographing device 10 through the communication unit 21.

Next, the control unit 24 acquires the rotation angle of the movable unit 12 of the photographing device 10 (step S12). That is, the rotation angle detected by the rotation angle detection mechanism provided in the movable portion 12 of the photographing device 10 is transmitted from the photographing device 10, and the control unit 24 acquires the rotation angle via the communication unit 21.

Next, the control unit 24 detects the movement of the photographing device 10 (step S13). Specifically, the control unit 24 detects that the shooting range of the camera module 16 has changed due to the movement of the entire shooting device 10 or the rotation of the movable unit 12. That is, the control unit 24 analyzes the photographed image acquired from the photographing device 10, and determines that the photographing device 10 has been moved when the photographed image changes significantly. More specifically, the control unit 24 determines the presence or absence of movement based on the captured image and the amount of time variation of the rotation angle of the movable unit. While the movement is continuously detected, the control unit 24 determines that the photographing device 10 is still moving, and when the photographed image and the change in the rotation angle of the movable unit 12 are completed, the movement is performed. It is preferable to determine that it has been completed.

If the movement of the photographing device 10 is not detected (step S13: No), the process ends. On the other hand, when the movement of the photographing device 10 is detected (step S13: Yes), the control unit 24 performs the application estimation process (step S14). Immediately after the start of the automatic setting process, that is, at the time of the first execution of the automatic setting process, the movement detection in step S13 is not performed, and the process proceeds to the application estimation process in step S14.

The application estimation process is a process of estimating the interaction method intended by the user. As a premise, it is assumed that the application corresponding to each interaction method and the installation state (placement method) of the photographing device 10 are associated in advance. Here, it is assumed that the above-mentioned four applications A1 to A4 are installed on the PC20. The relationship between each application and the installation state of the photographing device 10 will be described below.

(A1) Gesture Operation Application for Screen When executing this application, the photographing device 10 is arranged at a position where the screen of the display unit 23 of the PC 20 is photographed, as illustrated in FIG. 6 (A). More specifically, it is assumed that the photographing device 10 is arranged at a position several tens of centimeters away from the screen of the PC 20 such as on a desk so as to look up at the screen. In this case, the captured image of the photographing device 10 includes the screen of the PC 20.

(A2) Gesture operation application near the keyboard In this application, the user makes a gesture on the keyboard, and the photographing device 10 photographs the user's hand or finger at that time. Therefore, as illustrated in FIG. 6B, it is assumed that the photographing device 10 is arranged at the upper part of the screen of the PC 20 so as to look down on the keyboard area. In this case, the captured image of the photographing device 10 includes the keyboard.

(A3) Line-of-sight input application This application photographs the user's face and detects the line-of-sight direction based on the movement of the eyes (pupils). Therefore, as illustrated in FIG. 7A, it is assumed that the photographing device 10 is installed so as to look up at the face from the front of the person. When detecting the line of sight, it is common to install the photographing device 10 so as to look up from below in order to take a picture so that the pupil of the person is not blocked by the eyelids. Therefore, in this case, the photographing device 10 is fixed so that the photographing direction faces upward with respect to the position where the photographing device 10 itself is fixed, and the movable portion 12 of the photographing device 10 moves from the bottom to the top. The rotation angle is as if looking up.

(A4) Web camera application When using this application, the entire face of a person is photographed. Therefore, as illustrated in FIG. 7B, it is assumed that the photographing device 10 is arranged at an angle facing a person, such as on the upper part of the screen of the PC 20. In this case, the movable portion 12 of the photographing device 10 is a rotation angle for photographing the front surface, for example, a rotation angle in the state of FIG. 3 (A).

As described above, the installation state of the photographing device 10 differs depending on the application. Therefore, by the application estimation process, the control unit 24 will execute the interaction method assumed by the user, that is, the user, based on the captured image by the photographing device 10 and the rotation angle of the movable unit 12 of the photographing device 10. It is possible to determine the application to be used.

FIG. 8 is a flowchart of the application estimation process. First, the control unit 24 determines whether or not the captured image acquired from the photographing device 10 includes the screen of the PC 20 (step S21). Here, for example, infrared LEDs are set at the four corners of the screen of the PC 20, and it can be easily determined whether or not the screen of the PC 20 is included by taking a picture with the optical filter of the photographing device 10 turned on. And.

When the captured image includes the screen of the PC 20 (step S21: Yes), the control unit 24 estimates that the user intends to use the application A1 and selects the application A1 (step S22).

On the other hand, when the captured image does not include the screen of the PC 20 (step S21: No), the control unit 24 determines whether or not the captured image includes the keyboard of the PC 20 (step S23). In this case as well, as in the case of detecting the screen of the PC20, for example, infrared LEDs are installed at 6 to 8 places around the keyboard including the four corners, and the optical filter is turned on to take a picture. Can be detected. In this example, the screen of the PC 20 and the keyboard can be distinguished from each other by the number of infrared LEDs.

When the captured image includes a keyboard (step S23: Yes), the control unit 24 presumes that the user intends to use the application A2 and selects the application A2 (step S24).

On the other hand, when the captured image does not include the keyboard (step S23: No), the control unit 24 has an angle at which the photographing device 10 looks up from below based on the rotation angle of the movable unit 12 acquired from the photographing device 10. It is determined whether or not it is (step S25). For example, when the photographing device 10 is placed on the desk as shown in FIG. 7A, the rotation angle of the movable portion 12 is close to 180 degrees as shown in FIG. 3B. Therefore, when the rotation angle acquired from the photographing device 10 is close to 180 degrees, the control unit 24 can determine that the angle is looking up.

When the photographing device 10 is at an angle to look up (step S25: Yes), the control unit 24 presumes that the user intends to use the application A3 and selects the application A3 (step S26).

On the other hand, when the photographing device 10 is not at an angle to look up (step S25: No), the control unit 24 estimates that the user intends to use the remaining application A4 and selects the application A4 (step S27). ..

Then, the control unit 24 starts the application selected in any of steps S22, S24, S26, and S27 (step S28). In this way, the application estimation process is completed, and the process proceeds to step S15 of FIG.
In step S15, the control unit 24 makes application-specific settings according to the application selected in the application estimation process.

(1) When application A1 is selected in the application estimation process, the control unit 24 of the PC 20 sends a control signal to the photographing device 10 to turn on the infrared emission of the camera module 16 and turn on the optical filter. As a result, the camera module 16 can capture the reflection of infrared rays emitted from itself as a captured image while the application A1 is being executed, so that the captured image is relatively unaffected by the background and the user's finger or the like. Will be able to be detected.

Further, the control unit 24 acquires the position information of the screen from the captured image by utilizing the detection of the infrared LED. Then, the coordinate conversion parameter from the coordinates of the captured image to the coordinates with respect to the screen is calculated and set in the application A1. As a result, the control unit 24 can convert the position of the user's finger or the like detected from the captured image into the position with respect to the screen during the execution of the application A1.

(2) When application A2 is selected in the application estimation process, the control unit 24 of the PC 20 sends a control signal to the photographing device 10 to turn off the infrared light emission of the camera module 16 and turn off the optical filter. In the case of application A2, since it is known that the background of the area where the gesture is performed is the keyboard, it is possible to detect the user's finger or the like from the captured image with a relatively simple process. Specifically, the skin color region in the captured image can be detected as a user's finger or the like, or the keyboard image can be removed from the captured image as a known fixed background. Therefore, the functions of infrared light emission and optical filter are not required.

Further, the control unit 24 calculates the positions of the four corners of the keyboard based on the positions of the infrared LEDs and sets them in the application A2. As a result, the detection area of the gesture operation by the user can be limited to the keyboard. Further, by grasping the position of the keyboard (distance to the photographing device 10) in advance, when a finger is detected during the execution of the application A2, the position of the finger (distance to the photographing device 10) is calculated. It is possible to determine whether or not the finger is floating from the keyboard. This is useful for determining the start of a finger gesture.

(3) When application A3 is selected in the application estimation process, the control unit 24 sends a control signal to the photographing device 10 to turn on the infrared emission of the camera module 16 and turn on the optical filter. Further, the camera module 16 is set to give priority to resolution. As a result, the pupil reflecting the irradiated infrared rays can be photographed during the execution of the application A3. Further, by using a high-resolution image, the accuracy of detection in the line-of-sight direction can be improved.

(4) When application A4 is selected in the application estimation process, the control unit 24 sends a control signal to the photographing device 10 to turn off the infrared light emission of the camera module 16 and turn off the optical filter. As a result, while the application A4 is being executed, an image such as a user's face taken by the photographing device 10 can be photographed with visible light and transmitted to the other party of the videophone.

[Modification example]
(Modification example 1)
In the above application estimation process, when any of the applications A1 to A4 is selected, the control unit 24 automatically starts the application in step S28. Instead, the control unit 24 may not start the application but wait for the user to start the application. In this case, when the user starts the application, the control unit 24 may perform the setting for each application shown in step S15 of FIG. In addition, if the settings can be made before starting the application, the settings for each application may be made in advance before the user starts the application.

(Modification 2)
In the above embodiment, the positional relationship between the photographing device 10 and the PC 20 is calculated by detecting the LED provided at a predetermined position of the PC 20 from the photographed image or acquiring the rotation angle of the movable portion 12 of the photographing device 10. However, the installation state of the photographing device 10 may be detected by another method as described below.

As one method, the installation state of the photographing device 10 may be detected by analyzing the photographed image. Specifically, the screen and keyboard of the PC are detected according to the characteristics of the captured image. For example, the screen may be detected by detecting straight lines on four sides, or the keyboard may be detected by detecting a large number of rectangles. Alternatively, a screen display control means may be provided, a detectable marker may be displayed on the screen, the presence or absence of the marker and its position may be detected from the captured image, and the position of the photographing device 10 with respect to the screen may be calculated.

As another method, a sensor may be provided in the fixed portion 14 of the photographing device 10 to detect that an object is sandwiched or standing.

Further, the installation state of the photographing device 10 may be detected by using not only the relationship with the PC but also the positional relationship with the person. For example, it is assumed that the face is detected from the captured image and the position of the photographing device 10 with respect to the person can be grasped when the position / size of the person's face does not change or is small for a predetermined time (for example, 10 seconds) or more. In addition to the position and size of the face, the orientation may be detected. As a result, for example, the Web application may be selected when the face is facing forward and is larger than a predetermined size, and the line-of-sight input application may be selected when the face is looked up at an angle.

Further, the user may be notified of the detection status of the installation state of the photographing device 10. At least, when the movement of the photographing device 10 is detected, the detection status of the screen or keyboard from the photographed image may be displayed on the display unit 23 of the PC 20. For example, a captured image superimposed on the detection state of the LED or marker may be displayed on the display unit 23. As a result, the user can confirm whether or not the moved photographing device 10 includes the intended subject, and can finely adjust the position of the photographing device 10.

(Modification example 3)
In the above embodiment, the resolution, the infrared transmission / visible light cut filter are turned on / off, and the infrared light emission is turned on / off as the setting of the photographing device 10, and the position of the screen or keyboard with respect to the photographing device 10 is set as the application setting. Although the calculation is performed, it may have other functions and the setting may be performed from the automatic setting software.

Specifically, when the photographing device 10 includes mechanisms such as pan / tilt / zoom, these mechanisms are controlled according to the application. For example, in a line-of-sight input application, a person zooms in near the face to capture a high-resolution image around the eyes. On the other hand, when executing the gesture operation application for the screen, zoom out to capture the entire area where the gesture is performed.

Further, the range and direction of infrared light emission may be controllable. For example, when the line-of-sight input application is executed, infrared rays are irradiated mainly in the vicinity of the face, and when the gesture operation application for the screen is executed, infrared rays are widely irradiated.

Further, when using an application that does not require stereo shooting, it may be set to shoot with only one camera module 16.

Further, the exposure control method may be switched depending on the situation. For example, when the infrared transmission / visible light cut filter is turned on and infrared irradiation is performed for shooting, the exposure control may be suitable for subject detection by infrared shooting instead of the usual exposure control for visible light shooting.

10 Imaging device 11 Main body 12 Moving part 14 Fixed part 16 Camera module 20 PC
24 Control unit

Claims (1)

An acquisition unit that acquires at least one of first information that is information on an image captured by the photographing unit and second information that is information on a photographing range photographed by the photographing unit.
Among a plurality of applications that use the information of the image captured by the photographing unit, an application control unit that provides the user with an application corresponding to the information acquired by the acquiring unit.
An information processing device characterized by being equipped with.

Images