JP2021043705A - Information processing device, imaging device, information processing system, and input processing method thereof - Google Patents

Abstract

To provide an information processing device, an imaging device, an information processing system, and an input processing method thereof, which can improve operability.SOLUTION: An information processing device includes an acquisition unit 31 configured to acquire a reference image obtained by photographing a display surface of display means on which a predetermined marker is displayed from a device (wearable device) equipped with imaging means for enabling a user to capture an image, and a movement processing unit 32 configured to move the position of the marker on the display surface based on information obtained from the reference image.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to an information processing device, an imaging device, an information processing system, and an input processing method thereof.

By mounting a touch panel on the information processing device, intuitive operation becomes possible (for example, Patent Document 1).

Japanese Patent No. 6095810

However, a touch operation cannot be performed in an information processing device that does not have a touch panel. In such a case, the operation is indirectly performed using a mouse, a mouse pad, or the like, but it is difficult to achieve operability such as a touch operation.

The present disclosure has been made in view of such circumstances, and an object of the present disclosure is to provide an information processing device, an imaging device, an information processing system, and an input processing method thereof, which can improve operability. ..

The first aspect of the present disclosure is an acquisition unit that acquires a reference image obtained by photographing the display surface of the display means on which a predetermined marker is displayed from a device provided with the photographing means and capable of photographing by the user, and an acquisition unit obtained from the reference image. It is an information processing apparatus including a movement processing unit that moves the position of the marker on the display surface based on the information to be obtained.

A second aspect of the present disclosure includes a photographing unit capable of photographing, an acquisition unit that acquires a reference image obtained by photographing the display surface of the display means on which a predetermined marker is displayed, and a position of the marker on the reference image. , A calculation unit that calculates a target position that is a position of the display surface corresponding to the reference position on the reference image based on the reference position on the reference image, and a transmission unit that transmits information about the target position. It is an image pickup apparatus provided with.

A third aspect of the present disclosure is an information processing system including an information processing device that controls a predetermined marker displayed on the display surface of the display means, and a device provided with a photographing means and capable of photographing the display surface by a user. An acquisition unit that acquires a reference image obtained by photographing the display surface on which the marker is displayed, and a movement processing unit that moves the position of the marker on the display surface based on the information obtained from the reference image. It is an information processing system including.

A fourth aspect of the present disclosure is an acquisition step of acquiring a reference image obtained by photographing the display surface of the display means on which a predetermined marker is displayed from a device provided with the photographing means and capable of photographing by the user, and obtaining from the reference image. This is an input processing method in which a computer executes a movement processing step of moving the position of the marker on the display surface based on the information to be obtained.

According to the present disclosure, it is possible to improve the operability.

It is a schematic external view of the wearable device which concerns on one Embodiment of this disclosure. It is a schematic external view of the information processing apparatus which concerns on one Embodiment of this disclosure. It is a figure which showed the hardware configuration of the information processing apparatus which concerns on one Embodiment of this disclosure. It is a functional block diagram which showed an example of the function provided in the information processing system which concerns on one Embodiment of this disclosure. It is a figure which shows the example of a nested marker. It is a figure which shows the example of the movement processing of the information processing system which concerns on one Embodiment of this disclosure. It is a figure which shows the example of the coordinate system of the wearable device which concerns on one Embodiment of this disclosure. It is a flowchart about the marker movement processing executed by the information processing system which concerns on one Embodiment of this disclosure. It is a flowchart about the gesture determination processing executed by the information processing system which concerns on one Embodiment of this disclosure. It is a figure which showed an example of the image which concerns on one Embodiment of this disclosure.

Hereinafter, an embodiment of an information processing device, an imaging device, an information processing system, and an input processing method thereof according to the present disclosure will be described with reference to the drawings. In the present embodiment, the case where the information processing system is applied to the combination of the notebook PC and the ring-type wearable device 10 will be described as an example.

FIG. 1 is a schematic external view of a ring-type wearable device 10 according to an embodiment of the present disclosure. The wearable device 10 is equipped with a camera (shooting means) 11. As for the ring-type wearable device 10, a device such as a pen-type device can be used as long as it is a device provided with a camera (shooting means) 11. When a pen-type device is used, for example, a shooting means is provided at the pen tip.

As shown in FIG. 1, the wearable device 10 has a ring shape and can be worn as a ring on a user's finger as shown in FIG. The wearable device 10 can take a picture with the camera 11 while being worn on the finger and exchange information with the information processing device 1. The wearable device 10 can communicate with the information processing device 1 by using a wireless communication method such as Bluetooth (registered trademark). The communication may be a wireless method or a wired method.

The wearable device 10 is equipped with an acceleration sensor (gyro sensor), detects information related to the movement (operation) of the wearable device 10 (information related to the movement of the user's finger), and information processing device 1 and information. Can be exchanged. The wearable device 10 sequentially transmits the captured image (reference image) and the detection result of the acceleration sensor to the information processing device 1 by the transmission unit (communication unit) while being worn on the user's finger.

FIG. 2 is a schematic external view of the information processing device 1 according to the embodiment of the present disclosure. In the present embodiment, the information processing device 1 will be described as a notebook PC, but the information processing device 1 is not limited to the notebook PC, and may be, for example, an information processing terminal such as a desktop PC or a projector. The display means (display unit 5) may be provided integrally with the information processing device, or may be provided independently of the information processing device and displayed by the device. Further, the display may be performed by projection like a projector.

As shown in FIG. 2, the information processing device 1 includes, as an example, a main body-side housing 2 and a display-side housing 3, both of which are substantially rectangular parallelepipeds.

The main body side housing 2 includes an input unit 4. The input unit 4 is a user interface for the user to perform input operations, such as a keyboard composed of various keys for inputting characters, commands, etc., and a touch pad for moving a cursor on the screen and selecting various menus. Etc. are provided.

The display-side housing 3 includes a display unit (display means) 5 for displaying an image. In the present embodiment, the display unit 5 exemplifies an LCD (Liquid Crystal Display), but the display unit 5 may be not limited to an LCD but may be another display device such as an organic EL (Electroluminescence) display or a touch panel. The LCD displays various information according to the input display data on the display screen. As will be described later, the marker displayed on the screen of the display unit 5 is controlled.

Further, the display-side housing 3 includes a camera 7 and a light source 8 for acquiring an image. The camera 7 is arranged in the display-side housing 3 on the display surface side of the LCD at substantially the center above the display surface side, and is capable of photographing a subject in front (for example, a user's face). The light source 8 is arranged substantially in the center above the display surface side of the LCD in the display-side housing 3, and emits auxiliary light related to the shooting of the camera 7 to the subject in front.

The main body side housing 2 and the display side housing 3 are connected by a pair of left and right connecting portions 6 at their respective ends. The connecting portion 6 is a hinge and supports the main body side housing 2 and the display side housing 3 so as to be openable and closable.

FIG. 3 is a diagram showing a hardware configuration of the information processing device 1 according to the embodiment of the present disclosure. The information processing device 1 includes, for example, a CPU (Central Processing Unit) 21, a main memory 22, a storage unit 23, a display unit 5, an input unit 4, a communication unit 26, a camera 7, and a light source 8. I have. Each of these parts is directly or indirectly connected to each other via the bus 28, and performs various processes in cooperation with each other.

The CPU 21 controls the entire information processing device 1 by an OS (Operating System) stored in the storage unit 23 connected via the bus 28, and executes various programs stored in the storage unit 23. Various processes are executed by.

The main memory 22 is composed of a writable memory such as a cache memory and a RAM (Random Access Memory). The main memory 22 is used as a work area for reading the execution program of the CPU 21 and writing processing data by the execution program.

The storage unit 23 is, for example, a ROM (Read Only Memory), an HDD (Hard Disk Drive), a flash memory, or the like, and for example, information processing such as Windows (registered trademark), OS (registered trademark), Android (registered trademark), etc. It stores an OS for controlling the entire device 1, various drivers for operating peripheral devices in hardware, applications for specific tasks, and various data and files. In addition, the storage unit 23 stores programs for realizing various processes and various data required for realizing various processes.

The display unit (screen) 5 is a display means, for example, an LCD, which displays according to the control of the CPU 21.

In the present embodiment, the display means is provided in the information processing device 1 as the display unit 5, but the display means may be provided as a device independent of the information processing device 1. Further, the display may be performed by projection like a projector. That is, as long as the display is performed on the display surface, the display method is not limited.

The input unit 4 is, for example, a keyboard, a mouse, or the like, and is used by the user to perform an input operation.

The communication unit 26 is an interface for connecting to a network, communicating with other information processing devices, and transmitting / receiving information.

The camera 7 is controlled by the CPU 21 based on, for example, an input operation from the user.

The camera 7 is capable of taking pictures and has a lens, a lens driving unit, an image sensor, and the like. The lens captures the light from the subject and causes the image sensor to form an image of the subject. The image sensor converts the light captured by the lens into a signal charge and captures an image of the subject. The image signal from the image sensor is subjected to signal processing (image processing) by, for example, an ISP. The camera 11 may have the same configuration as the camera 7.

The light source 8 emits auxiliary light related to photography by the camera 7, and is controlled by the CPU 21.

In the present embodiment, various processes are performed on the information processing device 1 side, but part or all of the processing may be performed on the wearable device 10 side. In this case, the wearable device 10 may perform information such as the CPU. A configuration related to processing may be provided.

FIG. 4 is a functional block diagram showing an example of the functions included in the information processing system 30. As shown in FIG. 4, the information processing system 30 includes an acquisition unit 31, a calculation unit 34, a movement processing unit 32, and a processing execution unit 33 as main configurations.

Various processes, which will be described later, realized by each unit are executed by, for example, the CPU 21 of the information processing device 1. That is, it is realized by the CPU 21 of the information processing device 1 reading the program stored in the storage unit 23 into the main memory 22 and executing the program. The program stored in the main memory 22 or the storage unit 23 may be, for example, a program downloaded from a predetermined server or installed and saved from a predetermined recording medium, and is stored in the storage unit 23 at the time of manufacture. It may be a program that is installed. Various processes in the information processing apparatus 1 described later are realized by the function of the program.

In the present embodiment, the functions of each part of the information processing system 30 are provided in the information processing device 1, but some or all of the functions of each part may be provided in the wearable device 10. That is, since the functions of the information processing system 30 need only be achieved by the entire system, each function can be flexibly provided in each device constituting the system.

In the information processing system 30, the marker displayed on the screen of the display unit 5 is moved based on the reference image captured by the marker.

A marker is a sign displayed on the screen and acts as a cursor. As will be described later, the marker displayed on the screen moves on the screen in conjunction with the movement of the wearable device 10. The marker is, for example, a nested marker as shown in FIG. In FIG. 5, an example of a nested marker is shown on the left side, an enlarged version of the constituent elements constituting the nesting is shown in the center, and a further enlarged version (minimum element) is shown on the right side. By using a nested marker, the distance between the wearable device 10 and the screen can be estimated more accurately. The marker configuration is not limited to the nested type as long as the marker can recognize up, down, left, and right.

The acquisition unit 31 acquires a reference image obtained by photographing the display surface of the display means (display unit 5) on which a predetermined marker is displayed from a device (wearable device 10) provided with the photographing means and capable of photographing by the user. Specifically, when the screen is photographed by the wearable device 10 while the marker is displayed on the screen, the photographed reference image is transmitted from the wearable device 10 to the information processing device 1. In this way, a reference image (including a marker) of the screen taken by the wearable device 10 is acquired.

The calculation unit 34 and the movement processing unit 32 move the position of the marker on the display surface based on the information obtained from the reference image. Specifically, the display position of the marker is moved to a position on the screen corresponding to the reference position in the reference image based on the position of the marker and the reference position in the reference image. That is, the marker on the screen is moved so that the marker moves to the reference position on the reference image acquired from the wearable device 10. In the present embodiment, the case where the reference position, which is the target position of the movement of the marker, is the center position in the reference image will be described, but the position is not limited to the center position, and the position may be arbitrarily set in the reference image.

The calculation unit 34 calculates a target position, which is a position on the display surface corresponding to the reference position on the reference image, based on the position of the marker on the reference image and the reference position on the reference image. The calculation unit 34 acquires the reference image acquired by the acquisition unit 31. Then, the marker is specified in the reference image. Then, the calculation unit 34 sets a target position on the screen for moving the position of the marker based on the distance from the display surface to the shooting means (that is, the distance between the screen and the wearable device 10) when shooting is performed. Calculate. The distance between the screen and the wearable device 10 is estimated from the information on the size of the marker included in the reference image.

Based on the distance between the screen and the wearable device 10, the position and center position of the marker on the reference image, the display position of the marker on the actual screen, and the position on the screen corresponding to the center position of the reference image. Distance can be calculated. In addition, the tilt of the reference image with respect to the screen can be calculated from the orientation (tilt) of the marker. Therefore, the direction from the display position of the marker to the position on the screen corresponding to the center position of the reference image can be estimated. That is, the coordinates (target coordinates) of the position on the screen corresponding to the center position of the reference image can be specified.

For example, since the size of the display range of the reference image on the screen can be specified from the distance between the screen and the wearable device 10, the position of the marker on the reference image and the center position on the screen correspond to the center position of the reference image. Coordinates can be specified. The coordinates on the screen corresponding to the center position of the reference image may be specified by using, for example, the three-square theorem or the similarity ratio.

In this way, the movement processing unit 32 calculates the distance (that is, the coordinate position on the screen corresponding to the center position of the reference image) for moving the display position of the marker to the center position on the screen.

The movement processing unit 32 moves the position of the marker on the display surface to the target position. That is, the movement processing unit 32 moves the displayed marker to the target position specified on the screen. By moving the marker to the calculated coordinate position, the marker can be moved to the center of the reference image on the screen.

That is, the calculation unit 34 and the movement processing unit 32 move the marker on the screen following the movement of the wearable device 10.

FIG. 6 shows an example of the movement process. Each state of F1-3 indicates each stage related to the movement process. First, in F1, the marker is located at the center of the reference image by the previous processing. Then, when the wearable device 10 is moved, the position of the marker deviates from the center position as shown in the reference image of F2. Then, the coordinate position on the screen corresponding to the center position of the reference image is calculated from the display position of the marker on the actual screen and the position corresponding to the center position of the reference image. Then, as in F3, the marker is moved to a position (calculated coordinates) on the screen corresponding to the center position of the reference image. When the movement is performed, the process returns to F1 and the iterative process is executed. In this way, the marker is moved.

The moving method is not limited to the above as long as the display position of the marker can be moved to a position on the screen corresponding to the center position of the acquired reference image.

The process execution unit 33 executes a predetermined process at the display position of the marker based on the information related to the movement of the photographing means (camera 11) for capturing the reference image. Specifically, the processing execution unit 33 executes a swipe process or a click process based on the acceleration information in the axial direction set in advance in the photographing means (camera 11). In the present embodiment, a case where a swipe process and a tap process are performed as predetermined processes will be described, but the operations possible on the touch panel can be similarly applied.

The information related to the movement is information related to the movement of the wearable device 10 in the three-dimensional space. For example, acceleration information. In order to recognize the movement in the three-dimensional space, the wearable device 10 is set with a Cartesian coordinate system as shown in FIG. Specifically, the z-axis is set in the shooting direction (shooting direction). Then, the x-axis and the y-axis are set in the direction orthogonal to the z-axis. In other words, when the wearable device 10 is photographing the screen of the information processing device 1, the z-axis indicates the screen direction with respect to the camera 11. The movement of the wearable device 10 (that is, a gesture by the user) can be recognized by the acceleration in each axial direction in such Cartesian coordinates.

Specifically, in the processing execution unit 33, the wearable device 10 moves in the z-axis at a predetermined acceleration or higher, and moves in the axial direction (x-axis and / or y-axis) orthogonal to the z-axis at a predetermined acceleration or higher. If so, swipe processing is executed. The swipe process is a process that corresponds to the operation of sliding a finger while touching the screen on the touch panel, and moves the image displayed on the screen to the next image or the previous image (for example, moving the page of an electronic book). This is the process of switching.

In this way, when the user swipes, for example, the finger is moved from left to right. Further, when the user swipes, the finger may move in the direction of approaching or moving away from the screen. Therefore, in the processing execution unit 33, when there is a movement at a predetermined acceleration or more on the z-axis and there is a movement at a predetermined acceleration or more on the x-axis and / or the y-axis, it is estimated that the swipe operation is performed by the user. Then, the swipe process is executed.

In the swipe process, the process may be executed when the x-axis and / or the y-axis move at a predetermined acceleration or higher.

The process execution unit 33 taps when the wearable device 10 moves on the z-axis at a predetermined acceleration or higher and does not move in the axial direction (x-axis and / or y-axis) orthogonal to the z-axis at a predetermined acceleration or higher. Execute the process. The tap process is a process corresponding to an operation of touching the screen on the touch panel, and is a process of selecting or executing each function of the application at the display position of the marker.

In this way, when the user performs the tap operation, it is an operation of bringing the finger away from the screen to the screen. Further, when the tap operation is performed, there is little movement in the x-axis and / or y-axis directions. Therefore, in the processing execution unit 33, when there is movement at a predetermined acceleration or more on the z-axis and there is no movement at a predetermined acceleration or more on the x-axis and / or y-axis, it is estimated that the tap operation has been performed by the user. Then, tap processing is executed.

Next, the marker movement process executed by the information processing system 30 according to the present embodiment will be described. FIG. 8 is a flowchart showing processing performed by the information processing system 30 according to the present embodiment. The processing flow shown in FIG. 8 is executed when the information processing device 1 is activated and the wearable device 10 is used. For example, it is disclosed when an instruction to start an operation by a marker is input from the user.

First, the wearable device 10 is detected (S101). The detection is a process of detecting that the information processing device 1 and the wearable device 10 are connected. For example, they are connected to each other using Bluetooth® or the like.

Next, a marker is displayed on the screen, the entire screen is photographed by the wearable device 10, and it is determined whether or not the entire screen and the marker are detected in the reference image (S102). In S102, for example, a marker is displayed on the screen, and the user is notified to take a picture of the entire screen. Then, the reference images taken by the wearable device 10 are sequentially acquired, and it is determined whether or not the entire screen on which the marker is displayed is detected.

By photographing the entire screen as in S102, the distance between the screen and the wearable device 10 at the time of photographing the entire screen can be specified from the size of the screen in the information processing device 1 registered in advance. That is, since the correlation between the distance between the screen and the wearable device 10 and the size of the marker displayed on the screen can be obtained, in the subsequent processing, the reference image is determined by the size of the marker in the captured reference image. It is possible to grasp the distance between the screen at the time of acquisition and the wearable device 10.

If the size (pixel size) of the marker displayed on the screen is stored in advance, shooting may be performed in S102 so as to include the marker instead of the entire screen.

As described above, when the setting of the information processing device 1 and the wearable device 10 is completed, the marker is displayed on the screen so that the marker is located at the center position of the reference image captured by the wearable device 10 as an initial state. Then, the marker movement process is performed so as to follow the movement of the wearable device 10 (S103). That is, in S103, the display position of the marker is moved to the position of the screen corresponding to the center position in the reference image. Note that S103 is terminated when the user inputs an instruction to end the operation by the marker, or when communication with the wearable device 10 is terminated. That is, the marker can be operated during the execution of S103.

If the reference image sequentially acquired during the execution of S103 does not include the marker, S101 may be executed again.

Next, the gesture determination process will be described with reference to FIG. The flow shown in FIG. 9 is repeatedly executed at a predetermined control cycle when the automatic tracking process of the marker is performed. The automatic tracking process of the marker is performed by S103 in the flow of FIG.

First, it is determined whether or not the reference image contains only one marker (S201). The smallest element is an element of the smallest unit constituting the nested type as shown in FIG. 5 in the nested marker, and is stored in advance as the minimum element. That is, the case where the reference image contains only one marker is the case where it can be estimated that the wearable device 10 (that is, the user's hand) is sufficiently close to the screen. When the reference image contains only one smallest marker, the reference image contains the entire one minimum element and does not include the entire other minimum elements (the whole is included). It will be in a state (not fit). For example, when the reference image contains only one minimum marker, the reference image includes one of the entire minimum elements of the nested marker as shown in FIG.

In S201, since it is sufficient if it can be estimated that the user's hand has approached the screen (the distance between the wearable device 10 and the screen is less than a predetermined distance), it is determined by the size of the marker included in the reference image. May be good. In this case, for example, it may be determined whether or not the size of the marker in the reference image is equal to or larger than a predetermined area.

If the reference image contains only one marker (NO determination in S201), it cannot be estimated that the wearable device 10 is sufficiently close to the screen, and the process ends.

When the reference image contains only one minimum marker (YES determination in S201), it can be estimated that the wearable device 10 is sufficiently close to the screen. Therefore, it is determined whether or not the wearable device 10 has moved within a predetermined time (for example, within α seconds) at a predetermined acceleration or more (for example, β acceleration or more) (S202). That is, in S202, it is determined whether or not the wearable device 10 has made a quick movement (gesture). S202 is performed by acquiring the information of the acceleration sensor provided in the wearable device 10.

In S202, the predetermined time (for example, α) and the predetermined acceleration (for example, β) are set as initial settings based on the simulated operation of the user. For example, in the initial setting (the stage before the flow of FIG. 8 is started for the first time), how long is it when the user performs a simulated tap and swipe operation and the user taps and swipes in the simulated operation? Get information that shows how much acceleration is generated inside. Then, a predetermined time (for example, α) and a predetermined acceleration (for example, β) estimated to have been tapped or swiped are set based on the acquired simulated motion information. The information related to tap and swipe is not limited to the case of performing a simulated operation, and may be set based on the information (database) of another user or the like. That is, a fixed value may be set in advance.

When it is determined that the wearable device 10 has not moved at a predetermined acceleration or higher within a predetermined time (NO determination in S202), it cannot be estimated that the gesture has been performed, so the process ends.

When it is determined that the wearable device 10 has moved at a predetermined acceleration or higher within a predetermined time (YES determination in S202), it can be estimated that the gesture has been performed. Therefore, the gesture classification process is performed, and the process corresponding to the estimated gesture is executed. In the present embodiment, two types of gestures, swipe and tap, will be described, but other gestures may be applied.

First, it is determined whether or not the acceleration in the z-axis direction is equal to or greater than the threshold value (S203). That is, it is determined whether or not the wearable device 10 has quickly moved in the direction of approaching or moving away from the screen. The threshold value related to the acceleration in the z-axis direction may also be set based on the simulated operation as described above, or may be set based on the information (database) of another user or the like.

If the acceleration in the z-axis direction is not equal to or greater than the threshold value (NO determination in S203), the process ends.

When the acceleration in the z-axis direction is equal to or greater than the threshold value (YES determination in S203), it is determined whether or not the acceleration in the x-axis direction and / or the y-axis direction is equal to or greater than the threshold value (S204). In S204, if there is a movement above the threshold value in at least one of the x-axis direction and the y-axis direction, a YES determination is made. The threshold value related to the acceleration in the x-axis direction and / or the y-axis direction may be set based on the simulated operation as described above, or may be set based on the information (database) of another user or the like. Good.

When the acceleration in the x-axis direction and / or the y-axis direction is not equal to or greater than the threshold value (NO determination in S204), tap processing is performed (S205). In the tap process, the click may be performed at the display position of the marker, or the mouse cursor may be moved to the display position of the marker to perform the click.

In this way, the tap process can be performed as in the operation using the touch panel.

When the acceleration in the x-axis direction and / or the y-axis direction is equal to or greater than the threshold value (YES determination in S204), a swipe process is performed (S206). In the swipe process, the swipe may be performed at the display position of the marker, or the mouse cursor may be moved to the display position of the marker to perform the swipe. In the swipe process, the image displayed on the screen is switched to the next image or the previous image (for example, page movement of an electronic book).

In this way, it is possible to perform the swipe process as in the operation using the touch panel.

Next, the setting of the size of the displayed marker will be described. The marker plays the role of a cursor on the operation screen like a mouse cursor, and is used to estimate the distance between the screen and the wearable device 10. Specifically, as in S201 in the flow of FIG. 9, whether the screen and the wearable device 10 are close to each other by determining whether or not the marker included in the reference image is the smallest one. It is judged whether or not. By using the nested type as a marker, it is possible to estimate the distance between the screen and the wearable device 10 more accurately.

It is preferable to appropriately set the size of the nested marker so that the distance between the screen and the wearable device 10 can be appropriately estimated using the nested marker. For this purpose, for example, the size of the nested marker is set at the initial setting stage when the information processing system 30 is used. The initial setting may be performed before the flow of FIG. 8 is first performed, or may be performed before each time the flow of FIG. 8 is performed before S101 is performed.

In the initial setting, the user is notified to perform a simulated tap and swipe operation on the screen, and the size of the minimum element of the nested marker is set based on the simulated operation. Specifically, when the wearable device 10 approaches the screen by the simulated operation, the size of the minimum element is set so that the reference image captured by the camera 11 includes the entire minimum element. The size of the minimum element is, for example, an average size corresponding to a plurality of simulated movements. By setting the size of the minimum element in this way, when the wearable device 10 approaches the screen when tapping or swiping is performed as in a simulated operation, there is only one minimum element in the reference image. Since it can be included, it is possible to more accurately detect that a swipe or tap may have been performed. The entire nested marker is configured based on the set minimum elements.

When the marker is not a nested type and uses another shape (for example, a shape that can recognize up, down, left, and right), the size (display area) of the shape may be determined by a simulated operation. For example, when the wearable device 10 approaches the screen by a simulated operation, the size of the marker is set so that the reference image captured by the camera 11 includes a marker having a predetermined area or more.

The size of the marker is not limited to the case of performing a simulated operation, and may be set based on information (database) of another user or the like. That is, a fixed value may be set in advance.

As described above, according to the information processing device, the image pickup device, the information processing system, and the input processing method thereof according to the present embodiment, a predetermined marker is displayed based on the information obtained from the captured reference image. Since the display position of the marker in the means is moved, the marker can be moved according to the position where the reference image is taken. That is, the marker to be displayed can be moved in conjunction with the position where the reference image is taken. For example, if the wearable device 10 is used for shooting, the cursor can be operated by following the movement of the wearable device 10. That is, it is possible to perform an intuitive operation like a touch panel.

Further, since the marker moves following the movement of the wearable device 10, the user does not need to touch the screen. Therefore, even if the user is at a position away from the screen, it is possible to operate the marker on the screen. Therefore, the convenience of the user can be improved.

Further, since a small device such as a wearable device 10 can be used, the configuration can be made compact.

In the present embodiment, the marker is displayed on the notebook PC, but the marker may be displayed on the screen projected by the projector.

The present disclosure is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the invention.

For example, when the ring-type wearable device 10 is used, the wearable device 10 takes a picture while being worn on the user's finger (for example, the index finger). Therefore, there may be a difference between the fingertip and the position of the camera 11. Therefore, the reference position in the reference image, which is the target position of the marker movement, is corrected based on the difference between the fingertip and the camera 11 position, and the target position of the marker movement is corrected so as to approach the fingertip. Good.

Further, when the marker movement process is performed and an input interface such as a keyboard, mouse, or touch pad is used, the marker movement process may be terminated or the marker movement process may be performed.

Further, in the present embodiment, the combination of the ring-type wearable device 10 and the information processing device (notebook PC) 1 has been described as an example. However, the wearable device 10 is not limited to the ring type if the photographing means is provided, and other devices such as a pen type device can be used as long as the photographing means is provided and the user can perform a gesture. Further, if a display means (including a case of displaying on a screen or a case of projecting and displaying like a projector) and a photographed image can be acquired from the photographing means, the application is not limited to a notebook PC.

Further, each process of the information processing system 30 can be arbitrarily shared by the wearable device 10 and the information processing device 1. That is, as described as an embodiment, the information processing is not limited to the case where the information processing is performed on the information processing apparatus 1 side. For example, the processing may be performed on the wearable device (imaging device) 10 side. In such a case, the wearable device (imaging device) 10 acquires a photographing unit (camera 11) capable of photographing and a reference image obtained by photographing the display surface of the display means on which a predetermined marker is displayed. A calculation unit 34 that calculates the target position, which is the position of the display surface corresponding to the reference position on the reference image, based on the position of the marker on the reference image and the reference position on the reference image, and the target position. It suffices to include a transmitter for transmitting information about the image. The information regarding the target position is information related to the coordinates of the target position. Then, the display means is controlled, the position of the marker on the display surface is moved to the transmitted target position, and the automatic follow-up process is performed.

Further, for example, each process of the information processing system 30 may be arbitrarily shared by the wearable device 10 and the information processing device 1. That is, information processing including an information processing device 1 that controls a predetermined marker displayed on the display surface of the display means, and a device (wearable device 10) that includes a photographing means (camera 11) and can photograph the display surface by the user. In the system, the acquisition unit 31 that acquires a reference image obtained by photographing the display surface on which the marker is displayed, and the movement processing unit 32 that moves the position of the marker on the display surface based on the information obtained from the reference image. Should be provided. The calculation unit 34 may be provided in the information processing device 1 or the wearable device 10. Even with such a configuration, automatic tracking processing is performed.

1: Information processing device 2: Main unit side housing 3: Display side housing 4: Input unit 5: Display unit 6: Connecting unit 7: Camera 8: Light source 10: Wearable device 11: Camera 21: CPU
22: Main memory 23: Storage unit 26: Communication unit 28: Bus 30: Information processing system 31: Acquisition unit 32: Mobile processing unit 33: Processing execution unit 34: Arithmetic unit

Claims (10)

An acquisition unit that acquires a reference image obtained by photographing the display surface of the display means on which a predetermined marker is displayed from a device having a photographing means and capable of photographing by the user.
A movement processing unit that moves the position of the marker on the display surface based on the information obtained from the reference image.
Information processing device equipped with. A calculation unit that calculates a target position, which is a position on the display surface corresponding to the reference position on the reference image, based on the position of the marker on the reference image and the reference position on the reference image. Prepare,
The information processing device according to claim 1, wherein the movement processing unit moves the position of the marker on the display surface to the target position. The information processing apparatus according to claim 2, wherein the reference position is a central position in the reference image. The information processing device according to claim 2 or 3, wherein the calculation unit calculates the target position based on the distance from the display surface to the shooting means when the shooting is performed. The information processing device according to any one of claims 1 to 4, wherein the marker is a nested marker. The information processing apparatus according to any one of claims 1 to 5, further comprising a processing execution unit that acquires information related to the movement of the photographing means and executes processing according to the information related to the movement. The information processing apparatus according to claim 6, wherein the processing execution unit executes swipe processing or click processing based on acceleration information in the axial direction set in advance in the photographing means. A shooting unit that can shoot and
An acquisition unit that acquires a reference image of the display surface of the display means on which a predetermined marker is displayed, and an acquisition unit.
A calculation unit that calculates a target position, which is a position of the display surface corresponding to the reference position on the reference image, based on the position of the marker on the reference image and the reference position on the reference image.
A transmitter that transmits information about the target position and
An imaging device comprising. An information processing system including an information processing device that controls a predetermined marker displayed on a display surface of a display means, and a device that has a photographing means and can photograph the display surface by a user.
An acquisition unit that acquires a reference image obtained by photographing the display surface on which the marker is displayed, and an acquisition unit.
An information processing system including a movement processing unit that moves the position of the marker on the display surface based on the information obtained from the reference image. An acquisition process of acquiring a reference image obtained by photographing the display surface of the display means on which a predetermined marker is displayed from a device having a photographing means and capable of photographing by the user.
A movement processing step of moving the position of the marker on the display surface based on the information obtained from the reference image, and
The input processing method that the computer performs.

Images