JP2022121468A - Information processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically perform starting, setting, and the like of an application that cooperates with an imaging apparatus, on the basis of information obtained from the imaging apparatus.

SOLUTION: In an information processing apparatus, an acquisition section acquires at least one of first information which is information on an image captured by an imaging section and second information which is information on an imaging range to be captured by the imaging section. An application control section provides a user with an application corresponding to the information acquired by the acquisition section, out of a plurality of applications which use the information on the image captured by the imaging section. On the basis of the image obtained from the imaging section or information on the imaging range, an application to be executed in cooperation with the imaging section is automatically started or settings required for executing the application are automatically performed.

SELECTED DRAWING: Figure 5

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to a technique for automatically starting an application and making settings in a terminal device connected to a photographing device.

There are various applications that connect a camera to a PC and run using the captured image. For example, a videophone is one such example. Further, Patent Literature 1 discloses an application that detects hand movements of a PC user and enables operations (click, selection, execution, etc.) according to the detected hand movements.

Japanese Patent Publication No. 2010-500645

There may be multiple applications that can work in conjunction with a single camera. In this case, in order to switch from one application to another, the user must change the installation position of the camera, etc., then manually launch the application or configure the settings for that application. There was a problem that it was troublesome.

Examples of the problems to be solved by the present invention include the above. SUMMARY OF THE INVENTION It is an object of the present invention to automatically start and set an application that operates in cooperation with a photographing device based on information obtained from the photographing device.

According to a claim of the invention, there is provided an information processing apparatus comprising at least one of first information that is information about an image captured by a photographing unit and second information that is information about a photographing range photographed by the photographing unit. and an application control unit that provides a user with an application corresponding to the information acquired by the acquisition unit, among a plurality of applications that use information of the image captured by the imaging unit. characterized by

1 shows the configuration of an interaction system according to an embodiment of the present invention; It is a perspective view which shows the external appearance of an imaging device. The state of rotation by a movable part is shown. It is a block showing the functional configuration of the PC. 6 is a flowchart of automatic setting processing; An example of an installation state of an imaging device is shown. An example of an installation state of an imaging device is shown. 9 is a flowchart of application estimation processing;

In a preferred embodiment of the present invention, the information processing device stores at least one of first information, which is information about an image captured by a capturing unit, and second information, which is information about a capturing range captured by the capturing unit. and an application control unit that provides a user with an application corresponding to the information acquired by the acquisition unit, among a plurality of applications that use the information of the image captured by the imaging unit.

The above information processing apparatus acquires at least one of first information, which is information of an image taken by an imaging unit such as a camera, and second information about the imaging range, and provides an application corresponding to the acquired information to a user. offer. Therefore, based on the information about the image or the imaging range obtained from the imaging unit, it is possible to automatically provide the application executed in cooperation with the imaging unit. Here, providing an application includes at least one of starting the application and making necessary settings for the application.

In one aspect of the information processing apparatus described above, the application control unit provides a first application to the user when an information output device is included in the image captured by the imaging unit, and the imaging unit A second application is provided to the user when the information input device is included in the photographed image. In this aspect, different applications are provided depending on whether the captured image includes the information output device or the information input device.

In another aspect of the information processing device, the second information includes information about the shooting direction of the shooting unit, and the application control unit shoots an image above a position where the shooting unit is fixed. If so, the user is provided with the third application, and if the image above the position where the fixing portion is fixed is not being photographed, the user is provided with the fourth application. In this aspect, different applications are provided depending on whether the photographing unit is photographing upwards or downwards.

In another aspect of the information processing apparatus, the application control unit executes at least one of starting an application corresponding to the information acquired by the acquisition unit and setting the application. In still another aspect, the application control section performs settings for the imaging section when executing an application corresponding to the information acquired by the acquisition section. As a result, the application is automatically activated and necessary settings are performed based on the information about the captured image or the information about the imaging range.

In a preferred example of the information processing device, the photographing unit includes an infrared light emission function and an optical filter that transmits infrared rays and cuts visible light, and the application control unit turns on the infrared light emission function. and off, and on and off of the optical filter.

In another preferred example, the information processing device includes an information input device or an information output device, and the second information is a relative positional relationship between the photographing unit and the information input device or the information output device. The obtaining unit generates information indicating the relative positional relationship by image analysis of the first information.

In another preferred embodiment of the present invention, an information processing method executed by an information processing device connected to an imaging unit includes first information that is information of an image captured by the imaging unit; an acquiring step of acquiring at least one of second information that is information relating to an imaging range to be photographed by the imaging unit; and an application control step for providing the user with the application. This method also makes it possible to automatically provide an application that is executed in cooperation with the imaging unit based on the information about the image or imaging range obtained from the imaging unit.

In another preferred embodiment of the present invention, the information processing program executed by an information processing apparatus connected to the imaging unit and provided with a computer includes first information that is information of an image captured by the imaging unit; Acquisition means for acquiring at least one of second information that is information relating to an imaging range to be photographed by an imaging unit; The computer functions as application control means for providing corresponding applications to the user. By executing this program on a computer, the above information processing apparatus can be realized. This program can be stored in a storage medium.

Preferred embodiments of the present invention will be described below with reference to the drawings.
[Constitution]
FIG. 1 shows the configuration of an interaction system according to an embodiment of the present invention. The interaction system includes a photographing device 10 and a PC 20 as an information processing device. The photographing device 10 and the PC 20 are connected by wire using a signal cable or the like, or by wireless communication. In the interaction system, a photographed image photographed by the photographing device 10 is transmitted to the PC 20 . A plurality of applications that operate in cooperation with the photographing device 10 are installed in the PC 20 . When the user installs the photographing device 10 at a predetermined position and the photographed image is transmitted from the photographing device 10 to the PC 20, the PC 20, based on the received photographed image, receives the user's image from a plurality of applications installed therein. presumes the application to be executed, automatically starts the application, and automatically performs settings required for execution of the application.

FIG. 2 shows the appearance of the imaging device 10. As shown in FIG. The imaging device 10 includes a body portion 11 , a movable portion 12 and a fixed portion 14 . The fixing part 14 is a part for fixing the photographing device 10 in some way. In this embodiment, the fixing portion 14 is configured as a clip, and can be fixed by pinching it between the screen edges of the PC 20, or can be placed on the desk as a stand.

The movable portion 12 is a portion for the user to adjust the orientation and position of the body portion 11, and includes a rotating portion 12a, a supporting portion 12b, and a rotating shaft 12c. The rotating portion 12a is rotatable around the rotating shaft 12c with respect to the supporting portion 12b. The rotating portion 12 a is fixed to the main body portion 11 , and the support portion 13 is fixed to the fixed portion 14 . Thereby, the body portion 11 is rotatable with respect to the fixed portion 14 . In this embodiment, the rotating part 12 is configured to be capable of rotating only on one axis, but it may be provided with a multi-axis rotating mechanism or a sliding mechanism.

The body portion 11 includes a plurality of light emitting portions 15 and a pair of camera modules 16 . It is assumed that the main unit 11 has various photographing functions according to applications executed on the PC 20 side. In this embodiment, the main unit 11 is provided with a pair of camera modules 16 to form a stereo camera, and further includes three infrared LED light emitting units 15 . The camera module 16 is also provided with an infrared transmission/visible light cut filter (hereinafter simply referred to as an “optical filter”), not shown, which can be switched on/off. ON/OFF of the optical filter is switched by a control signal supplied from the PC 20 .

FIG. 3 shows how the movable part 12 rotates. FIG. 3A shows a state in which the main body 11 is erected substantially vertically with respect to the fixed part 14, and the camera module 16 of the main body 11 photographs in the direction of the arrow. FIG. 3B shows a state in which the main body 11 is rotated 180 degrees around the rotation axis 12c with respect to the fixed part 14, and the camera module 16 of the main body 11 rotates according to the rotation state (rotation angle). to shoot in the direction of the arrow. The movable portion 12 is provided with a rotation angle detection mechanism such as a rotary encoder (not shown), which detects the rotation angle of the rotating portion 12a in a state set by the user.

FIG. 4 is a block diagram showing the functional configuration of the PC 20. As shown in FIG. The PC 20 includes a communication section 21 , an input section 22 , a display section 23 and a control section 24 . The communication unit 21 transmits and receives control signals and captured images to and from the imaging device 10 by wired or wireless communication. The input unit 22 is an information input device such as a keyboard and a mouse, and in this embodiment, it is a keyboard and a mouse (not shown) provided in the main body of the laptop PC. The display unit 23 is an information output device such as a liquid crystal display, and in this embodiment is a liquid crystal display provided in the main body of the laptop PC.

The control unit 24 comprehensively controls the entire configuration of the PC 20, and is specifically configured by the CPU of the PC 20 and the like. The control unit 24 executes automatic setting software and camera applications (hereinafter simply referred to as “applications”) A1 to A4 that operate in cooperation with the camera.

The automatic setting software estimates an application corresponding to the interaction method that the user wants to use based on the image captured by the image capturing device 10, the state of the movable part 12 of the image capturing device 10, etc. It automatically performs settings necessary for the application.

The camera applications A1 to A4 are software that implements an interaction method provided by the interaction system, and are configured as follows in this embodiment.
(A1): Gesture operation application for the screen (A2): Gesture operation application near the keyboard (A3): Gaze input application (A4): Web camera application

Specifically, the application A1 is an application for operating menus, icons, and the like displayed on the screen of the display unit 23 by the user performing gestures such as pointing to the screen. Note that the gesture by the user in this case does not need to touch the screen with a finger, and can be performed in the air. This application enables touch panel-like operations. In addition, even if the finger or the like does not touch the screen, the position and trajectory of the finger can be acquired from the captured image, so interaction that cannot be achieved with a touch panel can be realized.

The application A2 is an application that is operated by the user making a gesture such as making a specific hand shape near the keyboard or waving a finger above the keyboard. With this application, you don't have to take your hands off the keyboard, so you can operate smoothly while typing.

Application A3 is an application that operates a pointer according to the user's line of sight. Specifically, the eye portion of the user is extracted from the photographed image of the user's face to detect the line-of-sight direction, and the operation input corresponding to the detected line-of-sight direction is validated.

Application A4 is an application such as a so-called videophone that performs a dialogue with a remote location with moving images. When the dialogue is not performed, only the image captured by the image capturing device 10 is displayed on the display unit 23 .

Note that each application can receive settings for itself from the auto-configuration software.

[Auto setting process]
Next, automatic setting processing by automatic setting software will be described. FIG. 5 is a flowchart of automatic setting processing. The automatic setting process is repeatedly executed by the control unit 24 such as the CPU of the PC 20 executing automatic setting software.

First, the control unit 24 acquires an image captured by the imaging device 10 (step S11). Specifically, the control unit 24 receives the captured image from the imaging device 10 through the communication unit 21 .

Next, the control section 24 acquires the rotation angle of the movable section 12 of the photographing device 10 (step S12). That is, the rotation angle detected by the rotation angle detection mechanism provided in the movable portion 12 of the imaging device 10 is transmitted from the imaging device 10 and the control section 24 acquires the rotation angle via the communication section 21 .

Next, the control unit 24 performs movement detection of the photographing device 10 (step S13). Specifically, the control unit 24 detects that the photographing range of the camera module 16 has changed due to movement of the entire photographing device 10 or rotation of the movable unit 12 . That is, the control unit 24 analyzes the captured image acquired from the imaging device 10, and determines that the imaging device 10 has been moved when the captured image changes significantly. More specifically, the control unit 24 determines the presence or absence of movement based on the captured image and the amount of temporal variation in the rotation angle of the movable unit. Note that while the movement is continuously detected, the control unit 24 determines that the photographing device 10 is still moving. Preferably, it is determined to be complete.

If the movement of the photographing device 10 is not detected (step S13: No), the process ends. On the other hand, when the movement of the imaging device 10 is detected (step S13: Yes), the control unit 24 performs application estimation processing (step S14). Immediately after starting the automatic setting process, that is, when the automatic setting process is executed for the first time, the movement detection in step S13 is not performed, and the process proceeds to the application estimation process in step S14.

Application estimation processing is processing for estimating the interaction method intended by the user. As a premise, it is assumed that the application corresponding to each interaction method and the installation state (placement method) of the photographing device 10 are associated in advance. Here, it is assumed that the aforementioned four applications A1 to A4 are installed in the PC 20. FIG. The relationship between each application and the installation state of the imaging device 10 will be described below.

(A1) Gesture Manipulation Application for Screen When executing this application, the photographing device 10 is placed at a position for photographing the screen of the display section 23 of the PC 20, as illustrated in FIG. 6A. More specifically, it is assumed that the imaging device 10 is arranged at a position several tens of centimeters away from the screen of the PC 20, such as on a desk, so as to look up at the screen. In this case, the captured image of the imaging device 10 includes the screen of the PC 20 .

(A2) Gesture operation application near the keyboard In this application, the user makes a gesture on the keyboard, and the imaging device 10 photographs the user's hands and fingers at that time. Therefore, as illustrated in FIG. 6B, it is assumed that the imaging device 10 is arranged above the screen of the PC 20 so as to look down on the keyboard area. In this case, the image captured by the image capturing device 10 includes the keyboard.

(A3) Line-of-sight input application This application captures the user's face and detects the line-of-sight direction based on the movement of the eyes (pupils). Therefore, as illustrated in FIG. 7A, it is assumed that the photographing device 10 is installed so as to look up from the front of the person. In the case of performing line-of-sight detection, it is common to install the photographing device 10 so as to look up from below in order to photograph a person's pupils without being blocked by eyelids. Therefore, in this case, the photographing device 10 is fixed so that the photographing direction faces upward with respect to the position where the photographing device 10 itself is fixed, and the movable part 12 of the photographing device 10 is moved from the bottom to the top. The angle of rotation is such that you can look up.

(A4) Web camera application When using this application, the entire face of a person is photographed. Therefore, as illustrated in FIG. 7B, it is assumed that the photographing device 10 is arranged at the upper part of the screen of the PC 20 or the like at an angle facing the person. In this case, the movable portion 12 of the photographing device 10 has a rotation angle for photographing the front, for example, the rotation angle in the state shown in FIG. 3(A).

As described above, the installation state of the imaging device 10 differs for each application. Therefore, in the application estimation process, the control unit 24 determines the interaction method assumed by the user, that is, the interaction method that the user intends to perform, based on the image captured by the image capturing device 10 and the rotation angle of the movable part 12 of the image capturing device 10 . It is possible to determine the application that is

FIG. 8 is a flowchart of application estimation processing. First, the control unit 24 determines whether or not the captured image acquired from the imaging device 10 includes the screen of the PC 20 (step S21). Here, for example, infrared LEDs are set at the four corners of the screen of the PC 20, and by taking an image with the optical filter of the imaging device 10 turned on, it is possible to easily determine whether or not the screen of the PC 20 is included. and

If the captured image includes the screen of the PC 20 (step S21: Yes), the control unit 24 presumes that the user intends to use the application A1, and selects the application A1 (step S22).

On the other hand, if the captured image does not include the screen of the PC 20 (step S21: No), the control unit 24 determines whether the captured image includes the keyboard of the PC 20 (step S23). In this case as well, similar to the detection of the screen of the PC 20, for example, infrared LEDs are installed at 6 to 8 locations around the keyboard, including the four corners, and the optical filter is turned on. can be detected. In this example, the screen and keyboard of the PC 20 can be distinguished from each other by the number of infrared LEDs.

If the captured image includes a keyboard (step S23: Yes), the control unit 24 presumes that the user intends to use the application A2, and selects the application A2 (step S24).

On the other hand, when the keyboard is not included in the captured image (step S23: No), the control unit 24 adjusts the angle at which the imaging device 10 is viewed from below based on the rotation angle of the movable part 12 acquired from the imaging device 10. (Step S25). For example, as shown in FIG. 7A, when the photographing device 10 is placed on a desk, the rotation angle of the movable portion 12 is close to 180 degrees as shown in FIG. 3B. Therefore, when the rotation angle acquired from the photographing device 10 is close to 180 degrees, the control unit 24 can determine that it is an upward angle.

If the photographing device 10 is at an upward angle (step S25: Yes), the control unit 24 presumes that the user intends to use the application A3 and selects the application A3 (step S26).

On the other hand, if the photographing device 10 is not at an angle to look up (step S25: No), the control unit 24 presumes that the user intends to use the remaining application A4, and selects the application A4 (step S27). .

Then, the control unit 24 activates the application selected in any one of steps S22, S24, S26, and S27 (step S28). Thus, the application estimation process ends, and the process proceeds to step S15 in FIG.
In step S15, the control unit 24 performs setting for each application according to the application selected in the application estimation process.

(1) When the application A1 is selected in the application estimation process, the control unit 24 of the PC 20 sends a control signal to the imaging device 10 to turn on the infrared emission of the camera module 16 and turn on the optical filter. As a result, the camera module 16 can capture the reflection of the infrared rays emitted from itself as a captured image while the application A1 is running. can be detected.

Furthermore, the control unit 24 acquires the position information of the screen from the photographed image using the detection of the infrared LED. Then, from the coordinates of the photographed image, coordinate conversion parameters for coordinates with the screen as a reference are calculated and set in the application A1. As a result, the control unit 24 can convert the position of the user's finger or the like detected from the captured image into a position relative to the screen while the application A1 is running.

(2) When application A2 is selected in the application estimation process, the control unit 24 of the PC 20 sends a control signal to the imaging device 10 to turn off the infrared emission of the camera module 16 and turn off the optical filter. In the case of application A2, since the background of the area where the gesture is performed is known to be the keyboard, the user's finger or the like can be detected from the captured image by relatively simple processing. Specifically, it is possible to detect a flesh-colored area in the captured image as the user's finger or the like, or remove the image of the keyboard from the captured image as a known fixed background. Therefore, functions such as infrared light emission and optical filters are not required.

Further, the control unit 24 calculates the positions of the four corners of the keyboard based on the positions of the infrared LEDs, and sets them in the application A2. Thereby, the detection area of the user's gesture operation can be limited within the keyboard. Furthermore, by grasping the position of the keyboard (distance from the imaging device 10) in advance, when the finger is detected during execution of the application A2, the position of the finger (distance from the imaging device 10) is calculated, It can be determined whether or not the finger is lifted from the keyboard. This is useful for determining the initiation of finger gestures.

(3) When the application A3 is selected in the application estimation process, the control unit 24 sends a control signal to the imaging device 10 to turn on the infrared emission of the camera module 16 and turn on the optical filter. Also, the camera module 16 is set to give priority to resolution. As a result, it is possible to photograph the pupil reflecting the irradiated infrared rays during execution of the application A3. Also, by using a high-resolution image, it is possible to improve the accuracy of detecting the line-of-sight direction.

(4) When application A4 is selected in the application estimation process, the control unit 24 sends a control signal to the imaging device 10 to turn off the infrared emission of the camera module 16 and turn off the optical filter. As a result, while the application A4 is running, an image of the user's face or the like captured by the imaging device 10 can be captured with visible light and transmitted to the other party of the videophone call.

[Modification]
(Modification 1)
In the application estimation process described above, when one of the applications A1 to A4 is selected, the control unit 24 automatically activates that application in step S28. Alternatively, the control unit 24 may wait for the user to start the application without starting the application. In this case, when the user activates the application, the control section 24 may perform the setting for each application shown in step S15 of FIG. Also, if the settings can be made before starting the application, the setting for each application may be performed in advance before the user starts the application.

(Modification 2)
In the above-described embodiment, the positional relationship between the photographing device 10 and the PC 20 is calculated by detecting the LED provided at a predetermined position of the PC 20 from the photographed image and obtaining the rotation angle of the movable portion 12 of the photographing device 10. However, the installation state of the photographing device 10 may be detected by other methods such as those described below.

As one method, the installation state of the imaging device 10 may be detected by analyzing the captured image. Specifically, the PC screen and keyboard are detected based on the features of the captured image. For example, the screen may be detected by detecting straight lines on four sides, or the keyboard may be detected by detecting many rectangles. Alternatively, screen display control means may be provided to display a detectable marker on the screen, detect the presence or absence of the marker and its position from the captured image, and calculate the position of the photographing device 10 with respect to the screen.

As another method, a sensor may be provided in the fixed portion 14 of the photographing device 10 to detect that an object is sandwiched or is standing.

In addition, the installation state of the imaging device 10 may be detected using the positional relationship with the person, not limited to the relationship with the PC. For example, it is assumed that the position of the photographing device 10 with respect to the person can be grasped when the position and size of the person's face do not change or are small for a predetermined time (for example, 10 seconds) after face detection is performed from the photographed image. Also, the orientation may be detected in addition to the position and size of the face. As a result, for example, when the face is facing forward and has a predetermined size or larger, the web application may be selected, and when the face is at an angle of looking up, the line-of-sight input application may be selected.

Also, the user may be notified of the detection status of the installation state of the imaging device 10 . At least, when the movement of the photographing device 10 is detected, the screen from the photographed image and the detection status of the keyboard may be displayed on the display unit 23 of the PC 20 . For example, the display unit 23 may display a photographed image superimposed on the detected state of the LED or marker. Thereby, the user can confirm whether or not the moved photographing device 10 includes the intended subject, and finely adjust the position of the photographing device 10 .

(Modification 3)
In the above embodiment, as the settings of the imaging device 10, the resolution, the on/off of the infrared transmittance/visible light cut filter, and the on/off of the infrared emission are performed. Although calculation is performed, other functions may be provided and settings may be performed from automatic setting software.

Specifically, when the photographing apparatus 10 has mechanisms such as pan/tilt/zoom, these mechanisms are controlled according to the application. For example, a gaze tracking application zooms in on a person's face to capture a high-resolution image around the person's eyes. On the other hand, when executing a gesture operation application on the screen, zoom out to photograph the entire area where the gesture is performed.

Also, the range and direction of infrared light emission may be controllable. For example, when executing a line-of-sight input application, infrared rays are emitted mainly around the face, and when executing a gesture operation application on the screen, infrared rays are emitted widely.

Also, when using an application that does not require stereo shooting, the setting may be such that only one camera module 16 is used for shooting.

Furthermore, the exposure control method may be switched depending on the situation. For example, when the infrared transmission/visible light cut filter is turned on and infrared irradiation is performed, exposure control suitable for subject detection by infrared photography may be performed instead of normal exposure control for visible light photography.

REFERENCE SIGNS LIST 10 photographing device 11 main body 12 movable part 14 fixed part 16 camera module 20 PC
24 control unit

Claims (1)

an acquisition unit that acquires at least one of first information that is information about an image captured by a capturing unit and second information that is information about a capturing range captured by the capturing unit;
an application control unit that provides a user with an application corresponding to the information acquired by the acquisition unit among a plurality of applications that use information of an image captured by the image capturing unit;
An information processing device comprising:

Images