JP7321459B2 - Mobile terminal, operation method, and program - Google Patents

Description

The present invention relates to an operation system, an operation method, and a program for operating designated equipment.

Non-Patent Document 1, for example, discloses a method of estimating an operator's line of sight through image recognition and operating an object (specified device) on the operator's line of sight.

Xucong Zhang, Yusuke Sugano, Andreas Bulling, Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications. .de/opengaze-toolkit-released)

With the technique disclosed in Non-Patent Document 1, for example, when operating a device placed on a table, a camera is required to photograph the operator's face and the device on the table at the same time. Also, multiple cameras are required to avoid blind spots and shading. A plurality of cameras may hinder the movement of the operator. In addition, the processing associated with time synchronization and coordinate conversion between cameras increases, which increases the cost of the operation system. As described above, the conventional operation method using the line of sight of the operator requires a plurality of cameras (equipment), and has the problem that the system configuration is complicated and the cost is high.

The present invention has been made in view of this problem, and it is an object of the present invention to provide an operation system, an operation method, and a program that can operate a specified device with a single mobile terminal.

An operation system according to an aspect of the present invention is an operation system comprising one or more devices and a mobile terminal that operates the devices, wherein the mobile terminal includes a camera that captures images of the devices and a camera that captures images with the camera. a device designating unit that acquires a device image and designates the device positioned at a reference point of the device image; and an operation signal generating unit that generates an operation signal for operating the designated device. .

Further, an operation method according to an aspect of the present invention is an operation method performed by the above operation system, comprising: a photographing step of photographing one or more devices to obtain a device image; The gist is to perform a device designation step of designating the device to be positioned, and an operation signal generation step of generating an operation signal for operating the designated device.

A program according to an aspect of the present invention is summarized as a program for causing a computer to function as the above operation system.

According to the present invention, a designated device can be operated with one mobile terminal. Therefore, the system configuration is simple and the cost can be reduced.

1 is a configuration diagram showing a configuration example of an operation system according to an embodiment of the present invention; FIG. 2 is a block diagram showing a functional configuration example of the mobile terminal shown in FIG. 1; FIG. 3 is a diagram schematically showing a device image of a device to be operated taken by the camera shown in FIG. 2; FIG. It is a figure which shows typically the side surface of the portable terminal shown in FIG. 5 is a diagram schematically showing an example of hemispherical images taken with the two fisheye lenses shown in FIG. 4; FIG. 6 is a diagram schematically showing an example of converting the two hemispherical images (omnidirectional images) shown in FIG. 5 into an equirectangular view. FIG. FIG. 7 is a diagram for explaining a method of extracting the line-of-sight direction of an operator from the equirectangular diagram shown in FIG. 6; 3 is a diagram schematically showing the configuration of a modified example of the camera shown in FIG. 2; FIG. FIG. 4 is a diagram schematically showing the principle of generating three-dimensional coordinate information from two device images and an operator image; 2 is a flow chart showing a processing procedure of an operation method performed by the operation system shown in FIG. 1; 1 is a block diagram showing a configuration example of a general-purpose computer system; FIG.

An embodiment of the present invention will be described below with reference to the drawings. The same reference numerals are given to the same items in multiple drawings, and the description will not be repeated.

FIG. 1 is a configuration diagram showing a configuration example of an operation system according to an embodiment of the present invention. An operation system 100 shown in FIG. 1 is composed of one or more devices 20 a , 20 b , 20 c and a mobile terminal 10 .

The example shown in FIG. 1 shows devices 20a, 20b, and 20c placed on a shelf 30 taller than a person, and an operator U operating these devices 20a, 20b, and 20c with a portable terminal 10. ing. Although an example in which the mobile terminal 10 is held by the operator U is shown, the mobile terminal 10 is not limited to this example. The mobile terminal 10 may be carried by any part of the operator U, such as the arm, shoulder, chest, leg, or the like.

Also, the devices 20a, 20b, and 20c may be placed at a position lower than the operator U, for example, on a table (not shown). FIG. 1 shows an example of positions that are relatively difficult for the operator U to operate.

The device 20a is, for example, a lighting whose illuminance can be adjusted by remote control. The device 20b is, for example, a radio that can be remotely controlled. The device 20c is, for example, a humidifier that can adjust the amount of humidification by remote control. Devices are not limited to this example. For example, it may be a self-propelled cleaner or the like arranged on the floor. In the following description, when it is not necessary to designate any one of the devices 20a, 20b, and 20c, the device 20 is simply referred to.

FIG. 2 is a block diagram showing an example of the functional configuration of the mobile terminal 10. As shown in FIG. As shown in FIG. 2 , the mobile terminal 10 has a camera 11 , a device designating section 12 and an operation signal generating section 13 .

The camera 11 is attached to the tip portion of the mobile terminal 10 . The operator U directs the tip toward the device 20 and takes an image of those devices. Camera 11 is a general digital camera.

The device designation unit 12 acquires the device image captured by the camera 11 and designates the device 20 located at the reference point of the device image. The reference point of the device image is, for example, center coordinates of the device image captured by the camera 11 .

An image frame (frame) having a predetermined number of pixels is set around the pixel of the central coordinates, and the device 20 is designated by recognizing the image within the image frame. For example, if the outer shape included in the image frame is spherical, the device 20a is specified, if the outer shape is a cube, the device 20b is specified, and if the outer shape is a cylinder, the device 20c is specified. Alternatively, the device 20 may be specified by recognizing a character string (device model, etc.) in the image frame.

The device designation unit 12 has information such as the external shape and device name, character string and device name, and designates the device to be operated from the device image. The external shape, device name, character string, device name, etc. are stored in advance in a memory (not shown).

The operation signal generator 13 generates an operation signal for operating the designated device 20 . The operation signal is generated by associating a switch signal obtained from a switch provided on the mobile terminal 10 with an operation specification of the device stored in a memory (not shown). When the device designation unit 12 designates the device 20b, the operation signal generation unit 13 generates an operation signal for increasing the volume of the device 20b (radio), for example, in correspondence with the switch signal of a predetermined switch.

The operation signal is converted into infrared rays (IrDA) or electromagnetic waves (WiFi, Bluetooth) and transmitted to the designated device 20 . In this way, the device 20 pointed at by the mobile terminal 10 can be designated and operated.

As described above, the operation system 100 according to the present embodiment is an operation system including one or more devices 20 and the mobile terminal 10 that operates the devices 20. The mobile terminal 10 shoots the device 20. a camera 11 that acquires a device image captured by the camera 11, a device designation unit 12 that designates a device 20 positioned at a reference point of the device image, and an operation signal that generates an operation signal for operating the designated device 20 and a generation unit 13 . As a result, the specified device 20 can be operated with one mobile terminal 10 . Also, since the number of components constituting the operation system 100 is small, the cost can be reduced.

(equipment image)
FIG. 3 is a diagram schematically showing an example of a device image. A device image OG shown in FIG. 3 is an image captured by the camera 11 of the mobile terminal 10 .

Assume that the equipment image OG is, for example, an image of 5 million pixels. In that case, the horizontal direction (x) of the equipment image OG is represented by 2560 pixels, and the vertical direction (y) is represented by 1920 pixels.

The reference point α shown in FIG. 3 is, for example, the center (x=1280, y=960) of the equipment image OG. Although the reference point α is shown in FIG. 3 for convenience of explanation, it is not necessary to display the reference point α. In addition, in FIG. 3, the reference point α is shown larger than the point with one pixel.

The device designation unit 12 sets an image frame (frame) having a predetermined number of pixels around the reference point α, and performs image recognition of an object included in the image frame. Therefore, which device 20 the operator U is pointing at can be detected from the reference point α and the image frame (not shown). That is, the operator U can designate the device 20 that he/she wants to operate. The size of the image frame may be automatically adjusted to include the object (device 20).

Next, an example in which the camera 11 is composed of a 360-degree camera will be described.

(mobile terminal with 360-degree camera)
FIG. 4 is a side view of the camera 11 of the mobile terminal 10 configured with a 360-degree camera 21. As shown in FIG.

As shown in FIG. 4, the 360-degree camera 21 is attached, for example, to the tip of a bar-shaped handle 15 . The handle 15 is a part that the operator U holds with his/her hand. Switches 131 and 132 operated by the operator U are provided on the surface of the handle 15 . A three-axis acceleration sensor 14 for detecting the zenith direction of the 360-degree camera 21 is provided inside the handle 15 . The triaxial acceleration sensor 14 adjusts the top and bottom of the omnidirectional image captured by the 360-degree camera 21 . The method of adjusting the top and bottom using the three-axis acceleration sensor 14 is well known.

The 360-degree camera 21 captures one 360-degree omnidirectional image by joining two hemispherical images captured by two fisheye lenses 211 and 212 . An imaging device (not shown) is provided at the central portion of the diameter of the fisheye lenses 211 and 212 .

A spherical image is an image obtained by photographing 360 degrees in both the vertical direction (z) and the horizontal direction (x). The depth direction (y) is included in the z-direction and x-direction omnidirectional images.

The fisheye lens 211 captures an equipment image OG (hemispherical image). The fisheye lens 212 captures an operator image SG (hemispherical image) of the operator U carrying the mobile terminal 10 .

FIG. 5 is a diagram schematically showing the equipment image OG and the operator image SG. The device image OG includes images of the shelf 30 and the devices 20a, 20b, and 20c placed thereon. The operator image SG includes an image of the upper body of the operator U holding the mobile terminal 10 with his left hand.

FIG. 6 is a diagram schematically showing a diagram in which an omnidirectional image captured by the 360-degree camera 21 is converted into an equirectangular diagram. The equirectangular diagram is a diagram obtained by converting the latitude and longitude of the omnidirectional image into xy coordinates. The conversion method is well known.

0 in the vertical and horizontal directions shown in FIG. 6 represents 0 degrees of latitude (φ) and 0 degrees of longitude (θ). λ=0 degree and φ=0 degree is the center position of the 360-degree camera 21 (A shown in FIG. 6). Latitudes of 0 to 90 degrees are elevation angles viewed from the 360-degree camera 21, and 0 to -90 degrees are depression angles viewed from the 360-degree camera 21. Longitude 0 to 180 degrees represents the x direction from the 360 degree camera 21, and longitude 0 to -180 degrees represents the -x direction from the 360 degree camera 21.

When the −x side vertex of the fisheye lens 211 is directed toward the device 20b (FIG. 1), the device 20b is located in the second quadrant above −90 degrees on the horizontal axis (C shown in FIG. 6). On the other hand, the operator U photographed with the fisheye lens 212 in that posture is located in front of the fisheye lens 212 and is located 90 degrees above the horizontal axis.

The viewpoint position (V _θ [deg], V _φ [deg]) of the operator U with respect to the 360-degree camera 21 can be obtained by using OpenPose, which is a technique for detecting feature points and detecting relationships between feature points. can be obtained. OpenPose (a pose estimation library) is well known.

Next, the line-of-sight direction (G _θ [deg], G _φ [deg]) based on the viewpoint position (V _θ [deg], V _φ [deg]) is acquired. The line-of-sight direction (G _θ [deg], G _φ [deg]) can be obtained by using OpenGaze that detects the line of sight. OpenGaze is well known.

Next, the position (L[20] _θ [deg], L[20] _φ [deg]) of the device 20 to be operated is acquired. The position (L[20] _θ [deg], L[20] _φ [deg]) of the device 20 can be obtained by using YOLO, which is an object detection method. YOLO (You Look Only Once) is well known.

_A gaze point _{( T θ [ deg} ] , T _φ [deg]) can be calculated.

FIG. 7 is a diagram schematically showing an example of a method of calculating the gaze point (T _θ [deg], T _φ [deg]) of the operator U. As shown in FIG. "A" shown in FIG. 7 is the position of the 360-degree camera 21, "B" is the viewpoint position (V _θ [deg]) of the operator U, and "C" is the device 20 to be operated. The calculation method will be explained in the elevation direction.

In FIG. 7, a circle is assumed centering on the position "A" of the 360-degree camera 21 and passing through the viewpoint position (V _θ [deg]) "B" and the device 20 "C". In this case, the distance from the 360-degree camera 21 to the device 20 is equal to the distance from the 360-degree camera 21 to the viewpoint position (V _θ [deg]) of the operator U.

Here, a straight line x' is assumed which has a position "A" of the 360-degree camera 21 as an origin and passes through a viewpoint position "B" parallel to the x-axis. Then, the angle at which the device 20 is viewed from the viewpoint position "B" is the line-of-sight direction (G _θ [deg]). Also, the angle formed by the line segment AB and the straight line x' is the same as the viewpoint position (V _θ [deg]).

Since the line-of-sight direction (G _θ [deg]) and the viewpoint position (V _θ [deg]) are known, the gaze point (T _φ [deg]) of the operator U can be calculated by the following equation.

The gaze point (T _θ [deg]) of the operator U in the longitude direction can be similarly calculated. Note that the gaze point (T _φ [deg]) may be represented by T _φ [deg]+V _φ [deg].

The device 20 that the operator U wants to operate is specified from the obtained point of gaze (T _θ [deg], T _φ [deg]). That is, the device 20 at (close to) the angle formed by the line segment AB and the line segment AC shown in FIG. 7 or the angle formed by the x-axis and the line segment AC is designated as the operation target.

As described above, the device specifying unit 12 detects the line-of-sight direction of the operator U from the operator image SG, and specifies, for example, the device 20b based on the line-of-sight direction. Also, the equipment image OG and the operator image SG may be obtained from omnidirectional image information captured by the 360-degree camera 21 . As a result, the specified device 20 can be operated with one mobile terminal 10 .

That is, if the operator U can specify the device 20 that he/she wants to operate, the operation signal generator 13 can easily generate an operation signal that matches the specifications of the device 20 .

Although the example of generating the operation signal based on the switch signal obtained from the operation switches 131 and 132 of the portable terminal 10 has been described, a method of generating the operation signal without using the operation switches 131 and 132 is also conceivable. In that case, the action of the operator U included in the operator image SG is recognized, and an operation signal corresponding to the action is generated. The gesture can be, for example, the number of blinks and the number of fingers. Thereby, the operation switches 131 and 132 of the mobile terminal 10 can be eliminated. By eliminating the operation switches 131 and 132, the cost of the exterior case of the mobile terminal 10 can be reduced.

Further, identification information for identifying the operator U may be included in the operation signal. Identification of the operator U can be performed by recognizing the face of the person holding the mobile terminal 10 . That is, the device designating unit 12 also obtains an operator image of the operator U carrying the portable terminal 10, recognizes the operator image and designates the operator U, and the operation signal generating unit 13 generates the operator U The operation signal includes identification information that identifies the As a result, the device 20 that has received the operation signal can provide a service corresponding to the operator U. FIG. For example, if the device 20 is a television, the service corresponding to the operator U is, for example, displaying the latest video program of the genre that the operator U has viewed in the past.

Note that the camera is not limited to the 360-degree camera 21 . The device image OG and the operator image SG may be captured by one general digital camera (camera 11).

FIG. 8 is a diagram showing a configuration of a modification in which one camera 11 captures the equipment image OG and the operator image SG. Two mirrors 40 a and 40 b are arranged facing the camera 11 . The device 20 is arranged in the -x direction of the mirror 40a, and the operator U is positioned in the x direction of the mirror 40b.

The device image OG and the operator image SG may be captured by the camera 11 configured as shown in FIG. The same effects as when the 360-degree camera 21 is used can be obtained.

Also, the device designation unit 12 may generate three-dimensional coordinate information from the two device images OG and the operator image SG.

FIG. 9 is a schematic diagram illustrating the principle of generating three-dimensional coordinate information from two equipment images OG and operator image SG. FIG. 9 shows a case where the operator U sticks out the mobile terminal 10 toward the device 20 from the state described in FIG. FIG. 4 is a hypothetical equirectangular diagram;

In this case, two isosceles triangles ABC and AB'C' centered on position "A" of the 360-degree camera 21 are obtained. Using these two viewpoint positions (V _θ [deg], V _φ [deg]) and the line-of-sight direction (G _θ [deg], G _φ [deg]), three-dimensional coordinate information can be obtained on the principle of a stereo camera. can.

In this way, the device designation unit 12 can generate three-dimensional coordinate information from the two device images OG and the operator image SG, and designate the device 20 by the three-dimensional coordinates. Thereby, for example, an operation signal can be selectively transmitted to a plurality of self-propelled cleaners placed on the floor.

(Method of operation)
FIG. 10 is a flow chart showing the processing procedure of the operation method executed by the operation system 100 according to this embodiment.

When the operation system 100 starts operating, the camera 11 captures one or more devices 20 to obtain a device image OG (step S1). When the camera 11 is a 360-degree camera 21, the operator image SG is also acquired at the same time.

The device designation unit 12 designates the device 20 located at the reference point α of the device image OG (step S2). The reference point α is, for example, central coordinates of an image captured by the camera 11 . An image frame (frame) having a predetermined number of pixels is set around the pixel of the central coordinates, and the device 20 is designated by recognizing the image within the image frame.

The operation signal generation unit 13 generates an operation signal for operating the device specified by the device specification unit 12 (step S3). The operation signal may be generated based on a switch signal obtained by operating a physical switch, or may be an operation signal corresponding to the action of the operator U recognized.

As described above, the operation method according to the present embodiment is an operation method performed by the operation system 100, and includes a photographing step S1 of photographing one or more devices 20 to acquire the device image OG, and a reference of the device image OG. A device designation step S2 of designating the device 20 positioned at the point α and an operation signal generation step S3 of generating an operation signal for operating the designated device 20 are performed. A designated device can be operated with one mobile terminal 10 .

The operation system 100 and the operation method can be realized by a general-purpose computer system shown in FIG. In such a computer system, each function of the operation system 100 except for the camera 11 is realized by executing a predetermined program loaded on the memory 91 by the CPU 90 . The prescribed program can be recorded on a computer-readable recording medium such as HDD, SSD, USB memory, CD-ROM, DVD-ROM, MO, etc., or can be distributed via a network.

According to the operating system 100 and the operating method according to this embodiment, the device 20 out of reach of the operator U can be operated. Also, the operation does not interfere with others. Also, the device 20 can be operated with less, for example, hand movement. In addition, there is no need for a head-up display or the like that causes discomfort when worn. Also, the operation right can be easily transferred among a plurality of persons.

The present invention is not limited to the above-described embodiments, and modifications can be made within the scope of the gist of the present invention. For example, although an example using the triaxial acceleration sensor 14 has been described for correcting the top and bottom of the omnidirectional image, the present invention is not limited to this example. For example, a marker can be placed in advance in the environment where the operation system 100 is placed, and the image of the marker can be recognized to correct the top and bottom of the omnidirectional image.

Also, although an example of recognizing an operator image to identify the operator U has been described, the present invention is not limited to this example. The operator U who holds the mobile terminal 10 may be specified based on the bone information detected by the above OpenPose, and the device 20 to be operated may be specified.

Also, although an example (FIG. 1) in which the device 20 to be operated is arranged on the shelf 30 is shown, the same applies to the device 20 placed on a table lower than the operator U's viewpoint, for example. can send operation signals to

Thus, the present invention naturally includes various embodiments and the like not described here. Therefore, the technical scope of the present invention is defined only by the matters specified in the scope of claims that are valid from the above description.

10: mobile terminal 11: camera 12: device designation unit 13: operation signal generators 20, 20a, 20b, 20c: device 21: 360-degree camera 30: shelves 40a, 40b: mirror

Claims (5)

A mobile terminal that operates one or more devices ,
a device designation unit that acquires an image captured by a camera that captures the device and the operator of the mobile terminal and specifies the device;
an operation signal generation unit that generates an operation signal for operating the specified device ,
The image is a 360-degree omnidirectional image obtained by joining the device image of the device captured by the first lens of the camera and the operator image of the operator captured by the second lens,
The device designation unit converts the 360-degree omnidirectional image into an equirectangular view, and uses the viewpoint position of the operator on the equirectangular view and the line-of-sight direction with the viewpoint position as a reference. Designating the device based on the calculated gaze point that the operator gazes at
A mobile terminal characterized by : The device designation unit
recognizing the operator image and designating the operator;
The operation signal generation unit
The mobile terminal according to claim 1, wherein identification information for identifying the operator is included in the operation signal. The operation signal generation unit
3. The portable terminal according to claim 1, wherein a gesture of the operator is recognized from the operator image, and the operation signal corresponding to the gesture is generated. An operation method performed by a mobile terminal that operates one or more devices ,
a device designation step of acquiring an image captured by a camera that captures the device and the operator of the mobile terminal , and designating the device;
an operation signal generation step of generating an operation signal for operating the specified device ;
The image is a 360-degree omnidirectional image obtained by joining the device image of the device captured by the first lens of the camera and the operator image of the operator captured by the second lens,
The device specifying step converts the 360-degree omnidirectional image into an equirectangular view, and uses the viewpoint position of the operator on the equirectangular view and the line-of-sight direction based on the viewpoint position. Designating the device based on the calculated gaze point that the operator gazes at
A method of operation characterized by A program for causing a computer to function as the portable terminal according to any one of claims 1 to 3 .

Images