JP2017212654A - Gui apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the efficiency of an operation to direct the visual line of a tracking camera to a target.SOLUTION: A wide area image display part 11 presents to a user a wide area image 110 including a target 3. A cursor generation part 12 causes a cursor to be displayed in the wide area image 110 presented to the user 1. A positional operation receiving part 13 receives an instruction from the user 1 to operate a position relating to the cursor in the wide area image 110. A target specification part 14 specifies the target 3 on the basis of information on the position of the cursor.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a GUI (Graphical User Interface) device, and more particularly to a GUI device suitable for a user to specify a target to be tracked by a so-called tracking camera.

  In order to continuously shoot an object, it is necessary to keep the camera's line of sight toward the object. In many cases, this is done manually by the photographer, but it is difficult to perfectly follow a fast and irregular movement such as a jumping ball. For this reason, research is being conducted on a system (so-called active vision: see Non-Patent Document 1 below) that automatically controls the camera's line-of-sight direction with a machine.

  In the normal active vision technology, the camera itself is attached to the driving head and moved, so that the response speed for movement in the line of sight is slow. With this, it is difficult to track a moving object (for example, a ball used in a ball game) including a rapid acceleration change. Considering the fact that the frame rate of high-speed cameras reaches 1 million fps at high speeds, and the current situation that image processing is accelerated by GPU, gaze control speed is a bottleneck in speed in various tracking systems. It can be said that

  In order to solve this problem, an optical system called a saccade mirror (Saccade Mirror) that changes the line of sight of a camera at high speed using a small drive mirror surface arranged in front of the camera has been proposed (Non-patent Document 2 and Patent Document below). 1). In this technique, the line of sight can be changed at high speed by using a biaxial galvanometer mirror. If the line of sight can be controlled so that the object is always captured at the center of the screen in the control system, it is considered that unprecedented dynamic imaging is possible. A camera that performs high-speed line-of-sight control is sometimes called a tracking camera, and an image (that is, an image of an object) acquired by the tracking camera is sometimes called a tracking image.

  Non-Patent Document 2 below also shows a method of keeping the tracking camera's line of sight toward an object using information (for example, color information) of an image captured by the tracking camera.

  By the way, in the above-described technique, it is necessary to specify the object by directing the line of sight of the tracking camera to the object in the initial stage of tracking. For this purpose, conventionally, a stopped object is used to direct the line of sight of the tracking camera to the object, and then a procedure for starting tracking is performed.

  However, in the case of this procedure, there is a problem that the work of directing the line-of-sight direction of the tracking camera toward the object is complicated. This procedure also has a problem that it is difficult to turn the tracking camera's line of sight on an object that has already started moving. For example, it is difficult to direct the tracking camera's line of sight to a moving ball after starting a ball game (for example, after a table tennis rally has started), assuming the above procedure.

JP 2015-14731 A

J. Aloimonos, I. Weiss and A. Bandyopadhyay: "Active Vision", Int'l Journal of Computer Vision, vol. 1, no. 4, pp. 333-356 (1988). K. Okumura, H. Oku and M. Ishikawa: "High-Speed Gaze Controller for Millisecond-order Pan / tilt Camera", Proc. Of IEEE Int'l Conf. On Robotics and Automation, pp. 6186-6191 (2011) .

  The present invention has been made in view of the above situation. One of the main objects of the present invention is to provide a technique capable of improving the efficiency of the work of directing the line of sight of a tracking camera to an object.

  Means for solving the above-described problems can be described as follows.

(Item 1)
A GUI device for designating an object to be tracked by a tracking camera,
A wide-area image display unit, a cursor generation unit, a position operation reception unit, and a target identification unit;
The wide area image display unit is configured to present a wide area image including the target to the user,
The cursor generation unit is configured to display a cursor in a wide area image presented to the user,
The position operation reception unit is configured to receive an instruction from the user for operating the position of the cursor in the wide area image,
The target specifying unit is configured to specify the target based on information on the position of the cursor.

(Item 2)
The GUI device according to item 1, wherein the wide area image is a moving image in real time about an environment where an object being tracked by the tracking camera exists.

(Item 3)
In addition, it has a designated operation reception unit,
The specific operation receiving unit is configured to receive an instruction from the user for specifying the object,
The target specifying unit is configured to specify the target based on information on the position of the cursor at the time when the designation operation receiving unit receives an instruction from the user, or after that. The GUI device according to 1 or 2.

(Item 4)
In addition, it has a wide area image reception unit,
The wide area image receiving unit is configured to receive the wide area image acquired by a wide area camera and send it to the wide area image display unit,
The GUI device according to any one of items 1 to 3, wherein the wide-area camera is configured to acquire the wide-area image by photographing the entire area to be photographed by the tracking camera.

(Item 5)
In addition, it has a tracking image display,
The GUI device according to any one of items 1 to 4, wherein the tracking image display unit is configured to display an image of the object photographed by the tracking camera.

(Item 6)
A method for a user to specify an object to be tracked by a tracking camera by means of a device,
Presenting a user with a wide area image including the object;
Displaying a cursor in the wide area image presented to the user;
Receiving instructions from the user to manipulate the position of the cursor in the wide area image;
Identifying the target based on information on the position of the cursor.

(Item 7)
A computer program for causing a computer to execute each step according to item 6.

  This computer program is stored in an appropriate recording medium (for example, an optical recording medium such as a CD-ROM or a DVD disk, a magnetic recording medium such as a hard disk or a flexible disk, or a magneto-optical recording medium such as an MO disk). Can be stored. This computer program can be transmitted via a communication line such as the Internet.

  According to the present invention, it is possible to efficiently perform the work of directing the line of sight of the tracking camera to an object.

It is a block diagram for demonstrating the schematic structure of the GUI apparatus which concerns on one Embodiment of this invention. It is explanatory drawing which shows an example of the presentation image shown to a user in the GUI apparatus of FIG. It is a flowchart for demonstrating the procedure in which a user specifies tracking object using the GUI apparatus of FIG. 4 is a flowchart for explaining a part of the procedure of FIG. 3 in more detail.

  Hereinafter, a GUI apparatus 10 according to an embodiment of the present invention will be described with reference to the accompanying drawings. This GUI device 10 is used to designate a target 3 to be tracked by the tracking camera 2 (see FIG. 1).

(Configuration of this embodiment)
The GUI device 10 according to the present embodiment includes a wide area image display unit 11, a cursor generation unit 12, a position operation reception unit 13, and a target identification unit 14 (see FIG. 1). Further, the GUI device 10 of this example includes a designation operation receiving unit 15, a wide area image receiving unit 16, and a tracking image display unit 17 as additional elements.

(Wide area image display)
The wide area image display unit 11 is configured to present a wide area image 110 (see FIG. 2) including the target 3 to the user 1 (see FIG. 1). Here, the entire image presented to the user 1 including the wide area image 110 (hereinafter referred to as “presented image”) is denoted by reference numeral 100 in FIG. The wide area image display unit 11 may be a functional element that can display the wide area image 110 for the user, and may be configured as a part of a display (not shown), for example.

  Here, the wide area image 110 in this embodiment is a moving image in real time about the environment where the target 3 tracked by the tracking camera 2 exists. For example, if the object 3 is a sphere used in table tennis, the environment may include a table tennis table, a racket, a player, a floor surface, a wall surface, and the like. However, the environment is not limited to these and may be any surrounding environment including the target.

(Cursor generator)
The cursor generation unit 12 is configured to display a cursor 120 (see FIG. 2) in the wide area image 110 presented to the user 1. In this embodiment, the cursor 120 is a rectangular frame, but there are no particular restrictions on the shape and size of the cursor. In short, it may be set so that the target can be specified.

(Position operation reception part)
The position operation receiving unit 13 is configured to receive an instruction from the user 1 for operating the position of the cursor 120 in the wide area image 110. Here, the position operation accepting unit 13 is an input / output device such as a mouse, for example, but there is no particular limitation as long as the cursor 120 can be adjusted to a necessary position on the image.

(Target specific part)
The target specifying unit 14 is configured to specify the target 3 based on the position information of the cursor 120. More specifically, the target specifying unit 14 of the present embodiment selects the target 3 based on information on the position of the cursor at the time when the designation operation receiving unit 15 receives an instruction from the user, or at a later time. It has a specific configuration. Detailed operation of the target specifying unit 14 will be described later.

(Designated operation reception part)
The designation operation receiving unit 15 is configured to receive an instruction from the user 1 for specifying the target 3. Here, the instruction from the user 1 for specifying the target 3 is, for example, the left click of the mouse in the wide-area image or the depression of the tracking button 220 (described later) in the presented image 100. There is no limitation, and an appropriate instruction input method can be set according to the application.

(Wide area image reception department)
The wide area image receiving unit 16 is configured to receive the wide area image 110 acquired by the wide area camera 4 (see FIG. 1) and send it to the wide area image display unit 11.

  The wide area camera 4 is configured to acquire the wide area image 110 by photographing the entire area to be photographed by the tracking camera 2 (that is, the environment where the target 3 exists).

(Tracking image display part)
The tracking image display unit 17 is configured to display an image of the target 3 taken by the tracking camera 2. Similarly to the wide area image display unit 11, the tracking image display unit 17 can be configured as a part of a display (not shown), for example.

  Here, the tracking camera 2 includes a camera body that can acquire an image at a high frame rate, a biaxial galvanometer mirror (so-called saccade mirror) that can change the camera's viewing direction at high speed, and a pupil transfer optical system. (Not shown). The biaxial galvanometer mirror includes a tilt mirror that rotates in the tilt direction and a pan mirror that rotates in the pan direction. The pupil transfer optical system can transfer the pupil position of the camera body between the tilt mirror and the pan mirror. As such a tracking camera 2, for example, the one shown in Non-Patent Document 2 can be used, and detailed description thereof is omitted.

(Operation of this embodiment)
Next, the operation of the GUI device according to the present embodiment will be described with further reference to FIG.

(Step SA-1 in FIG. 3)
First, the wide area image 110 acquired by the wide area camera 4 and the line-of-sight direction of the tracking camera 2 are calibrated. That is, it is in a state in which mutual conversion between the coordinates on the image in the wide area image 110 and the line-of-sight direction of the tracking camera is possible. Thereby, in the present embodiment, for example, when a certain pixel in the wide area image 110 is designated on the image, the tracking camera 2 can be directed in the corresponding line-of-sight direction. Conversely, in the present embodiment, the cursor 120 can also be displayed at a pixel position corresponding to the direction of the tracking camera 2.

(Step SA-2 in FIG. 3)
Next, the wide area camera 4 acquires the wide area image 110. The wide area image 110 is usually a moving image for the environment including the target 3. The acquired wide area image 110 is sent to the wide area image display unit 11.

(Step SA-3 in FIG. 3)
Next, the wide area image display unit 11 displays the wide area image 110 on a user terminal (not shown) such as a display for the user (see FIG. 2).

(Step SA-4 in FIG. 3)
Next, the user 1 designates a tracking target while viewing the wide area image 110. This tracking target designation operation will be described with further reference to FIG.

(Step SB-1 in FIG. 4)
First, the cursor generator 12 displays the cursor 120 in the wide area image 110 (see FIG. 2). In the initial state, the position of the cursor 120 in the wide area image 110 is arbitrary. For example, the cursor 120 can be displayed at the center of the wide area image 110.

(Step SB-2 in FIG. 4)
Next, the user adjusts the position of the cursor 120 by the position operation receiving unit 13. For example, it is assumed that the object 3 enters the cursor 120 (see FIG. 2).

  Next, the user inputs an instruction for specifying the target 3 through the designation operation receiving unit 15. This input is sometimes referred to herein as “cursor activation”. In this embodiment, for example, a method of left-clicking the wide area image 110 or pressing down the tracking button 220 can be used as an instruction for specifying the target 3.

(Steps SB-3 and SB-4 in FIG. 4)
Next, when an instruction for specifying the target 3 is input (that is, when the cursor becomes active), the target specifying unit 14 determines whether or not a tracking target exists in the cursor at that time. This determination can be performed using color information, for example.

  More specifically, for example, if the color of the target object is set to yellow, and the yellow area exceeds the specified area in the cursor, the center of the color area is set as the center of the target object (tracking position). Can be identified. However, an algorithm for specifying a tracking target and a tracking position and a threshold condition for specification can be appropriately set according to the application. For example, when the table tennis ball is the target 3, the target 3 can be specified by placing the cursor 120 on the ball held by the player before serving.

  Here, in the GUI device 10 of the present embodiment, if there is no target that satisfies the threshold condition in the cursor, the determination on the moving image (wide area image) is repeated until the threshold condition is satisfied at that position. In this way, tracking can be started when the object 3 passes the cursor 120 in the wide area image. This method is particularly effective when the target 3 has already started to move or when the target 3 is not visible at the present time and the trajectory of the target 3 can be predicted. For example, when it is certain that the ball passes through the net, such as table tennis and tennis, the cursor 120 is placed in a state of waiting for the ball by placing the cursor 120 near the net (see FIG. 2). Can do. Similarly, if the cursor 120 is placed in front of the player, tracking can be disclosed from the start of play (the time when the ball is supplied forward). In this way, the actual tracking can be started by specifying the position of the object 3 after the activation.

(Step SA-5 in FIG. 3)
When the target is specified by the above procedure, a tracking command including the tracking position is sent from the target specifying unit 14 to the tracking camera 2, and tracking to the target is started. In tracking, image-based tracking can be performed using an image (tracking image) obtained by the tracking camera 2. That is, using the tracking image information (for example, the positional deviation information of the target with respect to the image obtained immediately before), the direction of the tracking camera 2 is controlled to track the target 3, and the image of the target 3 is continuously acquired. Can do. As such a tracking method, for example, the method described in Non-Patent Document 2 described above can be used, and detailed description thereof will be omitted.

(Step SA-6 in FIG. 3)
On the other hand, the cursor generation unit 12 changes the position of the cursor 120 according to the position of the target 3 so that the cursor 120 is always displayed at a position surrounding the target 3. In the present embodiment, the cursor 120 can be displayed at a position surrounding the target 3 in the wide area image 110 using the correspondence relationship between the direction of the tracking camera 2 and the wide area image 110. Thereby, the user can easily recognize the position to be tracked and that the tracking is continued from the wide area image 110.

  In the present embodiment, the color of the cursor is changed between when tracking is continued and when tracking is interrupted. Thereby, the user can recognize the tracking state more clearly. Further, in this embodiment, when the object is lost after the cursor is activated (the object no longer exists in the cursor), the movement of the cursor can be stopped and the color of the cursor can be changed.

(Step SA-7 in FIG. 3)
Furthermore, the tracking camera 2 of the present embodiment can send the tracking image to the tracking image display unit 17 and display the tracking image 130 in the presented image 100. Since this tracking image is an image from the tracking camera 2 that continues to capture the target, even if the target 3 continues to move, the target 3 is almost stopped on the tracking image.

(Step SA-8 in FIG. 3)
The user can specify the target again by clicking the wide area image 110 again. If it is designated again, the process returns to step SA-5 to repeat the above operation. When stopping the tracking, for example, it can be stopped by an appropriate instruction means such as dragging the cursor 120 or pushing down the tracking button 220.

(Description of other parts in the presented image)
Hereinafter, another part of the presented image presented to the user in the GUI device of the present embodiment will be supplementarily described with reference to FIG.
Binarized image 140: Real-time binarized image obtained by image processing on the tracking image. Binarization threshold value input slider 150: A slider for adjusting a threshold value in a parameter for binarized image processing. This slider can be used to set a threshold value for specifying the target 3.
Frame rate 160: Frame rate of the tracking image input from the tracking camera 2 to the system side (GUI device side) Image processing time 170: Image processing for one frame of the tracking image (image processing for calculating the tracking direction) Time / frame drop counter 180 required for processing: number of frames lost per second in tracking image processing / tilt direction mirror angle 190: biaxial galvanometer mirror (so-called saccade mirror) for controlling the viewing direction of the tracking camera Mirror angle in the tilt direction (indicates the object position in the tilt direction)
Pan direction mirror angle 200: Mirror angle in the pan direction in the biaxial galvanometer mirror for controlling the line-of-sight direction of the tracking camera (indicating the position of the object in the pan direction)
Screen lock button 210: Button for switching lock / unlock of screen operation Tracking button 220: Button for instructing start and stop of tracking of target by tracking camera 2 Mirror control button 230: Control of saccade mirror Button for instructing start / stop of the target color information reset button 240: A button for resetting color information of a target specified as a target. By matching the button color with the color of the target 3, the user 1 can easily recognize the tracking target.
Recording button 250: Button for instructing start / stop of recording of tracking image (moving image). Wide area view switching button 260: Real-time image acquired by the wide area camera 4 as an image displayed as the wide area image 110. And a background acquisition button 270 for switching between the background image acquired in advance and the background image acquired before the tracking is started, or an image of the environment where the object should exist (background image) is acquired or acquired by the wide-area camera Button to reset the selected image. This button is used when a background image is used to extract the target 3.
Server button 280: A button for displaying a communication state to the server to which a part of the functions of the GUI device 10 should be distributed or for instructing or shutting down the server. Setup button 290: Operation of the GUI device 10 Button to transition to the setup screen (not shown) for setting

  The configuration of the presented image 100 described above is merely an example, and an appropriate configuration is possible as long as the GUI function can be exhibited. Moreover, each part of the presentation image 100 can be displayed on a different display, or the presentation image 100 can be displayed in combination with other images.

  According to the above-described GUI device of the present embodiment, there is an advantage that the user 1 can efficiently perform the work of directing the line of sight of the tracking camera 2 toward the target 3 by operating the cursor 120.

  Further, the GUI device according to the present embodiment has an advantage that it is possible to easily know where the tracking camera 2 is looking in the environment from the position of the cursor 120 in the wide area image 110.

  The contents of the present invention are not limited to the above embodiment. In the present invention, various modifications can be made to the specific configuration within the scope of the claims.

  For example, the cursor 120 described above may have a distorted shape corresponding to the distortion of the wide area image 110. That is, the wide area image 110 has a certain amount of distortion corresponding to the angle of view of the wide area camera 4. Corresponding to this distortion, by making the cursor shape distorted, the specific work of the object 3 can be further facilitated.

  In the above-described embodiment, the target 3 is specified based on the color information. However, the target 3 is not limited to the color, and the target 3 can be specified using, for example, a shape or other feature amount.

  Furthermore, the above-described functional elements only need to exist as functional blocks, and may not exist as independent hardware. As a mounting method, hardware or computer software may be used. Furthermore, one functional element in the present invention may be realized by a set of a plurality of functional elements, and a plurality of functional elements in the present invention may be realized by one functional element.

  Further, the functional elements may be arranged at physically separated positions. In this case, the functional elements may be connected by a network. It is also possible to realize functions or configure functional elements by grid computing or cloud computing.

DESCRIPTION OF SYMBOLS 1 User 2 Tracking camera 3 Object 4 Wide area camera 10 GUI apparatus 11 Wide area image display part 12 Cursor generation part 13 Position operation reception part 14 Designation operation reception part 14 Target specification part 15 Designation operation reception part 16 Wide area image reception part 17 Tracking image display Section 100 Presented Image 110 Wide Area Image 120 Cursor 130 Tracking Image 140 Binary Image 150 Binary Threshold Input Slider 160 Frame Rate 170 Image Processing Time 180 Frame Drop Counter 190 Tilt Direction Mirror Angle 200 Pan Direction Mirror Angle 210 Screen Lock Button 220 Tracking button 230 Mirror control button 240 Target color information reset button 250 Record button 260 Wide area view switching button 270 Background acquisition button 280 Server button 290 Doo-up button

Claims (7)

A GUI device for designating an object to be tracked by a tracking camera,
A wide-area image display unit, a cursor generation unit, a position operation reception unit, and a target identification unit;
The wide area image display unit is configured to present a wide area image including the target to the user,
The cursor generation unit is configured to display a cursor in a wide area image presented to the user,
The position operation reception unit is configured to receive an instruction from the user for operating the position of the cursor in the wide area image,
The target specifying unit is configured to specify the target based on information on the position of the cursor. The GUI device according to claim 1, wherein the wide area image is a moving image in real time about an environment where an object being tracked by the tracking camera exists. In addition, it has a designated operation reception unit,
The specific operation receiving unit is configured to receive an instruction from the user for specifying the object,
The target specifying unit is configured to specify the target based on information on a position of the cursor at a time when the designation operation receiving unit receives an instruction from the user or a time after that. Item 3. The GUI device according to Item 1 or 2. In addition, it has a wide area image reception unit,
The wide area image receiving unit is configured to receive the wide area image acquired by a wide area camera and send it to the wide area image display unit,
The GUI device according to any one of claims 1 to 3, wherein the wide area camera is configured to acquire the wide area image by capturing an entire area to be captured by the tracking camera. In addition, it has a tracking image display,
The GUI device according to any one of claims 1 to 4, wherein the tracking image display unit is configured to display an image of the object photographed by the tracking camera. A method for a user to specify an object to be tracked by a tracking camera by means of a device,
Presenting a user with a wide area image including the object;
Displaying a cursor in the wide area image presented to the user;
Receiving instructions from the user to manipulate the position of the cursor in the wide area image;
Identifying the target based on information on the position of the cursor.   The computer program for making a computer perform each step of Claim 6.

Images