JP2005332061A - Operation unit, operation input terminal, operation method, and operation program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an operation unit capable of intuitive and easy-to-understand operation input. <P>SOLUTION: The operation unit 9 for inputting operation, based on the travel of the position of a camera device 10 includes a photographing means for acquiring images photographed by the camera device 10; an operation detection means for detecting the travel of the position of the camera device 10 based on the images; and an operation outputting means for outputting operation correlated with the travel of the detected position. The operation detection means performs forward travel detection in which the camera device 10 travels in a direction for approaching an object to be photographed, when an image is enlarged in time series, and performs backward travel in which the camera device 10 travels in a direction moving away from the object to be photographed, when the image is shrunk in time series. The operation outputting means outputs operation corresponding to the travel. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an operation device, an operation input terminal, an operation method, and an operation program.

  Conventionally, the operation of a portable terminal or the like is mainly performed by pressing a button. In addition, due to restrictions on the size of the terminal, the number of buttons is very small compared to the keyboard, and the buttons themselves are also small. For this reason, in order to operate a portable terminal, a user must repeat a button press operation, which is very inconvenient.

As a technique for reducing this button pressing operation, there is “a mobile phone input method and a mobile phone” by the applicant of the present application (Patent Document 1). This is to reduce the user's button press operation during text input or terminal operation by recognizing the spoken word from the movement of the user's lips during the operation of the mobile terminal. If this technique is used, the button press operation at the time of operation of a portable terminal can be reduced.
Japanese Patent Laying-Open No. 2002-252682 (paragraph [0007])

  However, conventional techniques such as Patent Document 1 have a problem that operations such as a button pressing operation and a mouth moving operation are not intuitive and difficult to understand. For example, when a button is pressed, a plurality of operations are assigned to one button. Therefore, the user needs to accurately grasp the correspondence between the button and the operation, and there is a difficulty in operability. It was. Similarly, when the user moves the mouth, the user is forced to bear a burden of grasping how to move the mouth and the operation associated with the moving method.

  SUMMARY OF THE INVENTION Accordingly, it is a primary object of the present invention to provide an operating device that solves the above-described problems and enables intuitive and easy-to-understand operation input.

In order to solve the above problem, the operation device according to claim 1 is an operation device that inputs an operation based on a movement of a position of a camera device,
The operating device is associated with a photographing unit that acquires an image photographed by the camera device, an operation detecting unit that detects a movement of the position of the camera device based on the image, and a movement of the detected position. Operation output means for outputting the operation being performed, and
When the image is enlarged in time series, the operation detection means detects the camera device as a pre-movement that moves in a direction approaching the shooting target, and the image is reduced in time series. , Detected as a movement after the camera device moves away from the object to be photographed,
The operation output means outputs at least one of an operation corresponding to the forward movement and an operation associated with the backward movement.

  Thereby, the user can efficiently input an operation by an intuitive and easy-to-understand operation input method of moving the position of the camera device.

The operation device according to claim 2 is the operation device according to claim 1, wherein the operation device accesses a hierarchical data structure,
The operation associated with the previous movement is an operation of accessing the data structure to a lower hierarchy,
The operation associated with the post-movement is an operation for accessing the data structure to a higher hierarchy.

  As a result, the user can access data having a hierarchical structure such as a Web page as if it were a stacked document, so that an access operation can be performed intuitively.

  The operation device according to claim 3 is the operation device according to claim 1 or 2, wherein the operation detection means is based on movement information of the position of the feature point extracted from the image. An enlargement or reduction of the image is detected.

  Thereby, by acquiring the movement of the position of the camera device from the captured image of the camera device, it is possible to effectively use existing assets such as a mobile phone having the camera device.

  The operation device according to claim 4 is the operation device according to claim 3, wherein the feature points are located within a predetermined selection window from a set of all feature points extracted from the image. It is characterized in that it is constructed by partially extracting a set of feature points.

  Thereby, feature points with high reliability are selected by the selection window, and image processing is performed based on the selected feature points, so that the accuracy of image processing is improved.

The operation device according to claim 5 is the operation device according to any one of claims 1 to 4, wherein the operation detection means is used when the image is translated in time series. Detect the direction of translation,
The operation output means outputs an operation of moving a cursor in the parallel movement direction.

  Accordingly, the user can intuitively perform the operation by moving the camera device three-dimensionally.

The operation input terminal according to claim 6 is an operation input terminal that inputs an operation based on the movement of the position of the camera device,
The operation input terminal includes a photographing unit that acquires an image photographed by the camera device, and an operation output unit that outputs an operation associated with the movement of the position of the camera device detected based on the image. , Including,
The movement of the position of the camera device includes at least one of a pre-movement in which the camera device moves in a direction approaching the object to be imaged and a post-movement in which the camera device moves in a direction away from the object to be imaged. ,
The operation output means outputs at least one of an operation corresponding to the forward movement and an operation associated with the backward movement.

  Thereby, the user can efficiently input an operation by an intuitive and easy-to-understand operation input method of moving the position of the camera device.

The operation input terminal according to claim 7 is the operation input terminal according to claim 6, wherein the operation input terminal performs access to a hierarchical data structure,
The operation associated with the previous movement is an operation of accessing the data structure to a lower hierarchy,
The operation associated with the post-movement is an operation for accessing the data structure to a higher hierarchy.

  As a result, the user can access data having a hierarchical structure such as a Web page as if it were a stacked document, so that an access operation can be performed intuitively.

The operation method according to claim 8 is an operation method for inputting an operation based on the movement of the position of the camera device,
The computer is associated with a shooting procedure for acquiring an image shot by the camera device, an operation detection procedure for detecting a movement of the position of the camera device based on the image, and a movement of the detected position. And an operation output procedure for outputting the operation being performed, and
The operation detection procedure detects when the camera device is moved in a direction approaching the object to be photographed when the image is enlarged in time series, and when the image is reduced in time series. , Detected as a movement after the camera device moves away from the object to be photographed,
The operation output procedure outputs at least one of the operations corresponding to the forward movement or the movements associated with the backward movement.

  Thereby, the user can efficiently input an operation by an intuitive and easy-to-understand operation input method of moving the position of the camera device.

The operation method according to claim 9 is the operation method according to claim 8, wherein the computer accesses a hierarchical data structure,
The operation associated with the previous movement is an operation of accessing the data structure to a lower hierarchy,
The operation associated with the post-movement is an operation for accessing the data structure to a higher hierarchy.

  As a result, the user can access data having a hierarchical structure such as a Web page as if it were a stacked document, so that an access operation can be performed intuitively.

  An operation program according to a tenth aspect causes a computer to execute the operation method according to the eighth or ninth aspect.

  Thereby, the user can efficiently input an operation by an intuitive and easy-to-understand operation input method of moving the position of the camera device.

  The present invention is characterized by intuitively performing an operation that had to press a selection button, a clear button, or the like by moving the camera back and forth. As an effect, when performing an operation of following the link of the Web page, the user performs an operation of moving the camera device back and forth, and the user views the Web page as if it existed in three dimensions. It becomes possible.

  In the present invention, the feature point selection window is applied when tracking the feature points extracted from the image, thereby reducing the calculation load for tracking.

  Furthermore, the zoom detection accuracy can be improved by changing the number of reference frames for zoom detection according to the movement distance of the feature point detected between the frames.

  Hereinafter, an embodiment of an operating device to which the present invention is applied will be described in detail with reference to the drawings. First, the configuration of the operating device according to the present embodiment will be described with reference to FIGS. 1, 2, 5, 6, and 7.

  As illustrated in FIG. 7, the operation device 9 according to the present embodiment is held by a user, for example, and the camera device 10 of the operation device 9 captures an image. The user moves the camera device 10 three-dimensionally to move the position of the camera device 10. Then, the controller device 9 detects the movement of the camera device 10 from the captured image.

  The moving direction of the camera device 10 that provides a captured image to the operation device 9 will be described with reference to FIG. FIG. 5 is a diagram in which the camera device 10 is arranged in a three-dimensional space. A three-dimensional space is represented by an X axis, a Y axis, and a Z axis. Here, it is assumed that the camera device 10 has a lens for photographing in the Z-axis direction.

  In this case, moving the camera device 10 back and forth indicates moving the camera device 10 in the Z-axis direction. Furthermore, the camera device 10 moving forward indicates that the camera device 10 approaches the object to be imaged, and the camera device 10 moving later indicates that the camera device 10 moves away from the object to be imaged. When the camera device 10 moves to the left and right, the camera device 10 moves in the direction of the X axis, and when the camera device 10 moves up and down, the camera device 10 moves in the direction of the Y axis. It shows that.

Here, when the user holds the camera apparatus 10, the movement range may be limited. For example, when the user wants to input a determination operation based on the previous movement, if the user's hand has already been extended, or if there is an obstacle at the movement destination of the camera apparatus 10, the camera apparatus 10 There are restrictions on physical movement.
Therefore, the camera device 10 may be provided with an operation validity period specifying means using buttons or the like as means for allowing the user to specify whether or not to accept an operation input by movement.
As a result, when the camera device 10 cannot move any further before, the user invalidates the operation by the operation validity period designating means, lowers the camera device 10 later, and again performs the operation by the operation validity period designation means. It becomes possible to move the camera apparatus 10 forward after enabling it. Therefore, even when the moving range of the camera is finite, it is possible to avoid restricting the number of operation inputs.

In the following description, moving the camera device 10 forward is defined as “zoom in”, and moving the camera device 10 backward is defined as “zoom out”.
Here, as means for changing the zoom, there are known analog zoom for adjusting the focus of the lens by hardware, and digital zoom for changing the scale of the photographed image by image processing. These analog zooms and digital zooms generally have a configuration in which the user operates by pressing a zoom button associated with these functions, and pressing the zoom button does not move the camera device 10 itself. It can be considered not accompanied.
Therefore, in order to clearly distinguish the physical movement of the apparatus main body from the activation of the zoom function by the zoom button, the camera device 10 has, for example, a configuration in which the operation valid period designation unit is linked to the zoom button. Also good. That is, the camera device 10 adds a signal for invalidating the operation to the image to be transmitted to the operation device 9 while the zoom button is pressed. When the invalid signal is detected, the controller device 9 does not detect zoom by image processing.
This makes it possible to more accurately detect the physical movement of the apparatus main body.

  Next, the operation device 9 to which the present invention is applied is characterized in that the movement of the position of the camera device 10 by the user is detected, and the operation corresponding to the movement direction is set as the operation output result. FIG. 6A is a diagram illustrating a correspondence between a Web page (P01 to P23) that is an example of the content to be presented and a user operation (such as left movement) using the camera device 10 to access the Web page. It is.

  For example, an operation for displaying a page P22 by following links in order from P01 which is the top page currently being browsed by the user will be described. First, the user moves the camera device 10 to the left, and moves the cursor (not shown) on the link character to the page P11. Next, the user moves the camera device 10 forward and selects (determines) the link to the P11 page selected by the cursor, thereby switching the display from the P01 page to the P11 page. . Further, the user moves the camera device 10 to the right, and moves the cursor over the link character to the page P22. Then, the user moves the camera device 10 forward and selects (determines) the link to the P22 page selected by the cursor, thereby switching the display from the P11 page to the P22 page.

  FIG. 6B is a diagram illustrating the effect of the operation device 9 to which the present invention is applied. The Web pages that are the contents are P01, P11, P21, P31, and P41 in order from the upper layer, and the layers are connected by links. The user can efficiently access these hierarchical structures from the upper level to the lower level and from the lower level to the upper level by moving the camera apparatus 10 back and forth. As a result, the user can access the Web page as if it were a stacked document, so that the access operation can be performed intuitively.

  The operation device 9 shown in FIG. 1 includes an operation input terminal 1 that receives an operation input and an operation analysis device 2 that analyzes the input operation. The operating device 9 may be configured by a mobile terminal such as a mobile phone or a PDA (Personal Digital Assistance). Accordingly, it is possible to allow the user to input various operations while covering the small number of operation buttons that are restrictions of the mobile terminal.

  The operation device 9 is configured as a computer including at least a memory serving as a storage unit used when performing arithmetic processing and an arithmetic processing device that performs the arithmetic processing. The memory is constituted by a RAM (Random Access Memory) or the like. Arithmetic processing is realized by an arithmetic processing unit configured by a CPU (Central Processing Unit) executing a program on a memory.

  The photographing unit of the operation device 9 includes a camera device 10 that photographs and creates an image, and an image acquisition unit 11 that transmits a photographing request to the camera device 10 and receives an image.

  The operation detection means of the operating device 9 includes a feature point information storage unit 20 that stores information on feature points included in the captured image, an image feature point extraction unit 21 that extracts feature points included in the captured image, and the like. A selection window application unit 22 that selects a feature point by applying a feature point selection window when the captured image is set as an initial frame; and a feature point detected from the frame of the captured image A feature point ID assigning unit 23 that assigns an ID for identification, a feature point information registering unit 24, a feature point tracking unit 25 that tracks feature points between consecutively photographed frames, and a feature point tracking result And a zoom detection unit 26 that detects a zoom operation of the image taken from.

  The operation detection means of the operation device 9 further includes a count storage unit 30 that stores the count of the number of frames of the image, a user ID information storage unit 31 that stores information for identifying the user of the operation input terminal 1, and an image The frame count unit 32 that counts the number of frames and the first reception after the image reception process is started or the image reception process is restarted by performing initialization during the image reception process. And an initial frame setting unit 33 that sets a frame as an initial frame.

  The operation output means of the controller device 9 includes an output setting storage unit 40 that stores correspondence information between an input operation and an operation to be executed, an information output unit 41 that outputs an operation specified from the input operation, And a display device 42 that displays the result reflecting the operation.

  The image acquisition unit 11 includes means for transmitting an image capturing command to the camera device 10, capturing images at a certain known time interval, and receiving the captured images. At this time, the photographing time interval may be 1/15 second or 1/30 second. Any time interval may be used as long as the operation device 9 can shoot. The camera device 10 may be integrated with the operation device 9 or may be connected by a cable or the like. Alternatively, it may be in a separated state for wireless communication with the camera device 10.

  The image feature point extraction unit 21 has means for acquiring feature points included in an image photographed by the camera device 10. At this time, as feature amounts extracted from the image, information such as luminance information, hue information, contours, and vertices may be acquired and used. At this time, not only a single feature but also a plurality of features may be used simultaneously. The image feature point detection unit extracts feature points from the entire image.

  The frame count unit 32 has means for counting the number of frames, and is initialized when the image received by the image acquisition unit 11 is an initial frame of processing or when tracking fails during processing. At this time, it is assumed that there is also means for initializing the count of the number of frames. The number of frames is defined as the number of images received from the camera device 10 for each image (frame) constituting the moving image transmitted from the camera device 10. The number of frames is used to calculate statistics (for example, the average moving direction) of feature points between a plurality of frames in order to obtain a stable detection result in the zoom detection unit 26. .

  The feature point tracking unit 25 has means for tracking feature points included in a frame, and has means for obtaining a change in position of each feature point. At this time, it is assumed that the feature point may be selected by assigning an ID to the feature point to facilitate tracking, or by applying a feature point selection window to the initial frame. In addition, when the feature point goes out of the screen and tracking fails, it may be possible to have means for determining initialization.

  Note that a template prepared in advance may be applied to the selection window, or the user may designate an area. Further, a method in which the computer calculates the selection window may be applied. The calculation method of the selection window is such that, for example, an image region that is to be imaged by the camera device 10 is out of focus with a region that matches a specific focus adjusted by the autofocus of the camera device 10 (the image is blurred). ) A method of dividing a region into regions and selecting a region suitable for a specific focus as a selection window can be given. According to this method, since the reliability of the feature points in the out-of-focus area is low, noise of these feature points can be reduced.

  The feature point information registration unit 24 has means for registering, in the feature point information storage unit 20, displacement information between frames of the feature points tracked by the feature point tracking unit 25. The displacement information is defined as the amount of movement of the position of the corresponding feature point between frames. Here, as information to be registered, an ID for identifying a feature point, an elapsed frame number from the initial frame, shooting time interval information, position information of each feature point in each frame, information on whether or not tracking is successful Information such as, etc. may be registered. The feature point information storage unit 20 has means for storing information input from the feature point information registration unit 24.

  The zoom detection unit 26 has means for acquiring information stored in the feature point information storage unit 20 and detecting whether the position of the camera has moved back and forth from the initial frame from the displacement information of the feature points. Here, if the camera position moves before the initial frame, it is assumed that the zoom-in has been performed, and conversely, if it is determined that the camera has moved behind the initial frame, it is assumed that the zoom-out has been performed. . That is, the zoom detection unit 26 is characterized by detecting the movement of the position of the camera device 10 from the image processing on the image captured by the camera device 10.

  The information output unit 41 includes means for outputting contents according to the settings stored in the output setting storage unit 40 when the zoom detection unit 26 determines that the zoom-in and zoom-out operations have been performed. To do.

  The output setting storage unit 40 includes means for storing settings for the contents output from the information output unit 41 when zoom-in and zoom-out are detected by the zoom detection unit 26. For example, when zoom-in is detected, it is assumed that the “return key” is pressed, and when zoom-out is detected, the operation when the “clear” button is pressed is performed. It is assumed that setting information such as is stored.

  The present invention will be described with reference to the conceptual diagram of FIG.

  Reference numeral S <b> 201 is an image captured using the camera device 10 attached to the operation device 9. The subject to be photographed is present around the camera and is a still life. The distance from the subject is not specified.

  When an image captured by the camera device 10 is input to the present device, feature points included in the image are extracted. As a feature point extraction method, luminance information, hue information, contour information, and the like may be used. It is only necessary to extract feature points that are stably detected.

  Of the extracted feature points, a feature point selection window is applied so that feature points existing near the center of the image can be selected. This is shown by reference numeral S202. In step S202, a window in which rectangles overlap with feature points included in an image is applied, and feature points included in the window are selected. This makes it possible to obtain a feature point distribution having a certain distance from the center of the image.

  At this time, the shape of the window is not limited to the rectangular shape as in S202 but may be circular. The width of the frame question may be set in advance, but may be changed depending on the number of feature points included in the selection window, the average luminance of the image, and the distribution status of the histogram.

  The feature points are detected on the entire screen except for the initial frame. Then, the feature point detected in the initial frame is tracked to obtain a change in the feature point position.

  Reference numerals S203 and S204 indicate a state in which the camera is moved back and forth and the position of the feature point is changed. Feature points before change are indicated by circles, and feature points of moving words are indicated by diamonds. When the camera moves backward, the size of the object included in the image becomes smaller as it moves, and the distribution of feature points becomes narrower than before the movement. In addition, when the camera moves forward, the size of the object in the image becomes larger as it moves, so that the feature point distribution becomes wider than before the movement.

  For detection of back-and-forth movement, the fact that the feature point distribution expands or narrows radially is used. When the area of a figure such as a rectangle that includes a feature point is larger or smaller than the figure area of the feature point in the initial frame, it may be determined that it has moved back and forth, or all feature points have moved The vector may be obtained, and it may be determined whether it is directed outward or inward compared to the feature point position in the initial frame, and forward / backward movement is detected.

  In order to cope with noise such as camera shake caused by the user holding the camera, an image of several frames is acquired for detecting the displacement of the feature point. The number of acquired frames is changed from the magnitude of the position change of the feature point between the frames. For example, when moving the camera slowly back and forth, there are many displacement noises such as camera shake, so there are many acquisition frames, and conversely, when moving the camera rapidly, even if the number of acquisition frames is small, Since the back-and-forth movement is easily detected, the number of frames may be reduced.

  When a camera movement is detected from the displacement of the feature point, an output corresponding to the back and forth movement is output. This output may be a certain output, or a specific output may be performed by a prior setting. In addition, the advance setting is, for example, a camera movement in which the previous movement is determined (enter), the subsequent movement is returned (return), and the forward / backward / left / right movement is the forward / backward / left / right movement of the cursor. This is information for associating movements with actions on the screen.

  The configuration of the operation device 9 has been described above. Next, operation | movement of the operating device 9 of this embodiment is demonstrated along FIG.3 and FIG.4, referring FIG.1, FIG.2, FIG.5, FIG.6 and FIG.

  First, description will be made with reference to the flowchart of FIG. 3 is the operation device 9.

  First, in the image request transmission process (S301), a shooting request is transmitted to the camera device 10. The photographing request may include information such as a photographing time interval and a photographed image size.

  In the image reception process (S302), an image captured with the content transmitted to the camera device 10 in the image request transmission process (S301) is received. Here, the camera device 10 may be built in the operation device 9 or may be configured to be able to communicate with the operation device 9 by wired communication or wireless communication. Note that wired communication is performed, for example, according to the USB (Universal Serial Bus) standard.

  The controller device 9 determines whether the image received in the image reception process (S302) is that of the initial frame. If the number of execution times of the image request transmission process (S301) is the first time, it is determined as an initial frame (S303, Yes), and a frame counter initialization process (S304) is executed. Here, the condition determined as the initial frame includes a case where the tracking of the feature point has failed. In the case of tracking failure, a counter initialization request is transmitted, so that it may be determined as an initial frame when the initialization request is received.

  In the frame count process (S305), the number of frames is counted. When the frame counter initialization process (S304) is executed, it is assumed that the count number has been initialized.

  Thereafter, feature points in the image are extracted by feature point detection processing (S308). The feature point extraction method may use luminance information, hue information, contour information, or the like. Alternatively, a method combining these pieces of information may be used, and the extraction method may be changed depending on the input image.

  If the frame count has been initialized (S309, Yes), the selection window application process (S310) is executed to apply the feature point selection window. Thereafter, a feature point ID assigning process (S311) is performed on the feature point selected in the feature point ID assigning process (S311), and an ID is assigned to each selected feature point.

  The feature point tracking process (S312) tracks the feature point assigned with the ID. With reference to the feature point information storage unit 20, the displacement information of the feature points is calculated.

  In the feature point information registration process (S314), the displacement information of each feature point, the tracking result, and the like are registered in the feature point information storage unit 20 based on the feature point ID. If it is determined in step S309 that the frame is an initial frame, feature point ID information, position information, and the like are registered and referred to for tracking in the next frame.

  If it is determined that the tracking has failed (S315, Yes), an initialization request process (S316) is executed. In the initialization request process (S316), the initialization request is sent to the image reception process (S302), and the time when the next image is received is regarded as the first reception.

  If the tracking is successful (S315, No), a zoom detection process (S317) for detecting zoom-in and zoom-out is executed. Here, referring to the frame count number, the temporal change of zoom-in and zoom-out is obtained, and the detection result is output when either operation is performed continuously for a certain frame. By setting a change over a certain period of time as a threshold value, it is possible to deal with minute changes such as camera shake.

  Further, by referring to the feature point information storage unit 20 and obtaining the amount of change between frames, the amount of change is obtained by determining whether zoom-in and zoom-out operations have been performed rapidly or slowly. The time for detecting the zoom may be changed according to the amount of change. For example, when the zoom-in / zoom-out operation is performed rapidly, the amount of change between frames is large, so it is easy to detect even if the number of frames for zoom operation detection is reduced. Can be reduced.

  Conversely, when the operation is slow, it is possible to deal with noise such as camera shake by increasing the number of frames for zoom detection. In addition, in the zoom detection process (S317), it is assumed that the moving direction of each feature point in a certain period of time may be detected using a statistic such as an average of the displacement of feature points between a plurality of frames.

  In the output process (S318), when the zoom-in and zoom-out operations are detected, the output associated with each operation is referred to the output setting storage unit 40 and the information output unit based on the previous settings. 41 outputs an operation to the display device 42. Then, in the determination of continuation, when an end command is given from the outside, an end selection is performed and the process ends.

  A case where the processing of the present invention is performed between a server and a client will be described with reference to FIG. 4 are the operation input terminal 1 (corresponding to the client) and the operation analysis device 2 (corresponding to the server). Here, one operation analysis apparatus 2 (server) may handle a plurality of operation input terminals 1 (clients). Therefore, an ID for identifying the terminal is attached to each operation input terminal 1. When communicating data with the operation analysis device 2, the ID for identifying the terminal is used for each communication packet. To be included.

  4 differs from FIG. 3 in that the role of the operation device 9 is separated into the operation input terminal 1 and the operation analysis device 2. Thereby, since the operation analysis apparatus 2 performs image processing collectively, the calculation burden of the operation input terminal 1 can be reduced. Therefore, the operation input terminal 1 only needs to install a program for realizing the processing of the operation input terminal 1 in a mobile phone or the like that can execute a simple program that has been widely used. You do n’t have to. Thereby, the operating device 9 according to the present invention can be spread at a low cost.

  In the image request reception process (S401), the camera apparatus 10 is requested to transmit an image and is received.

  In the frame count process (S402), the number of times the image is acquired is counted. The input image is regarded as every moving image frame, and the number of frames is counted.

  The count storage unit 30 is a storage unit that stores the counted number of frames, and the user ID information storage unit 31 is a storage unit that stores a user ID. The user ID is used for identifying the transmission source when data is transmitted to the server side.

  In the image and user ID information transmission process (S404), the image acquired in the image request reception process (S401) and the user ID are transmitted to the server side. The image transmitted to the server side is received in the image and user ID information reception process (S405). Here, the user ID sent together with the image information is recognized, and the user is identified. It is confirmed whether the image data received from the user ID is an initial frame.

  If it is an initial frame (S407, Yes), the counter of the frame counter is initialized in the count initialization process (S408). Then, the frame count is executed in the frame count process (S409). When the count is initialized in the count initialization process (S408), the count is initialized. The count information is stored in the count storage unit 30.

  In the feature point extraction process (S410), feature points in the image are extracted. At this time, the feature point extraction method may be a luminance distribution or hue information, and may be extracted using a plurality of image feature amounts. In addition, the feature amount used from the image information may be changed.

  When the count is initialized by the count initialization process (S408) (S412, Yes), the selection window is applied by the selection window application process (S413) in order to select the feature point. To do. This selection window may be a window in which rectangles are overlapped as described in 202 of FIG. 2 or may be a window in which circles are overlapped. Assume that the selected feature points only need to be distributed with a certain distance from the center.

  The feature point selected in the selection window application process (S413) is given an identification ID in the identification ID assignment process (S414). The feature point ID information, the feature point position information, the frame count number, and the like are stored in the feature point information storage unit 20. Based on the information stored in the feature point information storage unit 20, the feature points are tracked in the feature point tracking process (S416).

  By the feature point tracking process (S416), the displacement information of the feature points between the frames is obtained. This displacement information is stored in the feature point information storage unit 20 in the feature point information registration process (S418).

  At this time, if tracking fails and feature points cannot be tracked (S419, Yes), initialization request processing (S420) is executed so as to return to the initial conditions again. When the image is initialized in the initialization request process (S420), the count initialization process (S408) is executed in the same manner as when an image is input for the first time in the image and user ID information reception process (S405). The frame counter is initialized.

  When the tracking is successful (S419, No), a zoom detection process (S421) for detecting whether or not zooming has been performed is executed. A detection result indicating whether zoom-in or zoom-out has been detected is transmitted to the client side in a detection result transmission process (S422). At this time, information such as zoom-in, zoom-out, no detection, or failure is transmitted as the detection result to be transmitted.

  The detection result information transmitted to the client is received in the detection result reception process (S423), and it is determined whether or not the detection is successful. As a result, if the detection is successful (S424, No), an output corresponding to zoom-in and zoom-out is output. At this time, output is performed based on the output setting of the output setting storage unit 40. Whether the detection is successful or unsuccessful, an end / continuation command transmission process (S427) is executed.

  This end / continuation command is received in the server side end / continuation command reception process (S428). If an end command is obtained from the reception result, the server-side processing is ended (S429, No). With these processes, it is possible to detect the back-and-forth movement of the camera from the displacement of the feature points of the image.

  The present invention described above can be widely modified without departing from the spirit thereof as follows.

  For example, the correspondence information between the movement of the camera device 10 and the cursor operation is not limited to the correspondence information, and various correspondence information may be used. For example, the previous movement may be a return (return) action, and the subsequent movement may be a decision (enter) action.

It is a block diagram of the operating device regarding one Embodiment of this invention. It is a conceptual diagram of the process of the selection window regarding one Embodiment of this invention. It is a flowchart which shows the process of the operating device regarding one Embodiment of this invention. It is a flowchart which shows the process performed between the server and client regarding one Embodiment of this invention. It is a conceptual diagram of the movement of the camera apparatus regarding one Embodiment of this invention. FIG. 4 is a diagram illustrating access to a data structure having a hierarchical structure according to an embodiment of the present invention. It is a figure which shows the usage example of the operating device regarding one Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Operation input terminal 9 Operation apparatus 10 Camera apparatus 20 Feature point information storage part 21 Image feature point extraction part 22 Selection window application part 26 Zoom detection part 40 Output setting storage part 41 Information output part

Claims (10)

An operation device for inputting an operation based on the movement of the position of the camera device,
The operating device is associated with a photographing unit that acquires an image photographed by the camera device, an operation detecting unit that detects a movement of the position of the camera device based on the image, and a movement of the detected position. Operation output means for outputting the operation being performed, and
When the image is enlarged in time series, the operation detection means detects the camera device as a pre-movement that moves in a direction approaching the shooting target, and the image is reduced in time series. , Detected as a movement after the camera device moves away from the object to be photographed,
The operation output means outputs at least one operation among an operation corresponding to the forward movement or an operation associated with the backward movement. The operating device shall access a hierarchical data structure,
The operation associated with the previous movement is an operation of accessing the data structure to a lower hierarchy,
The operation device according to claim 1, wherein the operation associated with the backward movement is an operation of accessing the data structure to an upper hierarchy.   The operation device according to claim 1, wherein the operation detection unit detects enlargement or reduction of the image based on movement information of a position of a feature point extracted from the image. .   The feature point is configured by partially extracting a set of feature points located within a predetermined selection window from a set of all feature points extracted from the image. 3. The operating device according to 3. The operation detection means detects the direction of translation when the image is translated in time series,
The operation device according to any one of claims 1 to 4, wherein the operation output means outputs an operation of moving a cursor in the direction of the parallel movement. An operation input terminal for inputting an operation based on the movement of the position of the camera device,
The operation input terminal includes a photographing unit that acquires an image photographed by the camera device, and an operation output unit that outputs an operation associated with the movement of the position of the camera device detected based on the image. , Including,
The movement of the position of the camera device includes at least one of a pre-movement in which the camera device moves in a direction approaching the object to be imaged and a post-movement in which the camera device moves in a direction away from the object to be imaged. ,
The operation input terminal is characterized in that at least one of the operations corresponding to the forward movement or the movements associated with the backward movement is read out from the storage means and output. The operation input terminal shall access a hierarchical data structure,
The operation associated with the previous movement is an operation of accessing the data structure to a lower hierarchy,
The operation input terminal according to claim 6, wherein the operation associated with the backward movement is an operation of accessing the data structure to a higher hierarchy. An operation method for inputting an operation based on the movement of the position of the camera device,
The computer is associated with a shooting procedure for acquiring an image shot by the camera device, an operation detection procedure for detecting a movement of the position of the camera device based on the image, and a movement of the detected position. And an operation output procedure for outputting the operation being performed, and
The operation detection procedure detects when the camera device is moved in a direction approaching the object to be photographed when the image is enlarged in time series, and when the image is reduced in time series. , Detected as a movement after the camera device moves away from the object to be photographed,
The operation output procedure outputs at least one operation among an operation corresponding to the forward movement or an operation associated with the backward movement. The computer shall access a hierarchical data structure;
The operation associated with the previous movement is an operation of accessing the data structure to a lower hierarchy,
The operation method according to claim 8, wherein the operation associated with the backward movement is an operation of accessing the data structure to a higher hierarchy.   An operation program for causing a computer to execute the operation method according to claim 8 or 9.

Images