JP6244712B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program that display display information at an arbitrary position on an image, for example.

  In recent years, with the development of information communication technology, an image processing technology based on augmented reality (AR), in which visual display information is added to an image obtained by imaging a real space (external world) using a computer and displayed. Development is underway. The interface in the augmented reality technology uses a camera (may be referred to as a two-dimensional code) attached to an object in front of the user's eyes using a camera fixed in an arbitrary place or a camera that can move freely. After acquiring an image including a marker by photographing, it has a function of displaying, on a display, additional information related to the object and display information associated with the marker.

  Specifically, the image processing method based on augmented reality is based on a reference position determined from the four corners of the captured marker and a physical coordinate system based on the reference position. In addition, it is characterized by being superimposed and displayed on a PC on which a camera is mounted or a display of a portable terminal. With this image processing method, it is possible to associate display information with the display position of an object.

  Using augmented reality technology, a function for identifying a fault location when a fault occurs in an electronic device or the like and supporting a field work such as a fault repair work by a user is realized. For example, an object to be a work target or a marker having a specific shape feature is provided near the object, and display information is superimposed and displayed in real time on an image captured by the camera based on various information included in the marker. . For example, a technology for superimposing and displaying an internal image and an operation procedure of a copier associated with a paper jam occurrence position on a copier serving as an object to be worked in a repair work support for a paper jam failure of a copier has been proposed. Yes.

Here, it is assumed that the following problems may exist in the field work by the conventional method which does not use the conventional augmented reality image processing technique.
(1-1) Bring a large amount of paper-based work procedure manuals describing the work location and procedures to the work site, and select and refer to the necessary locations by the worker himself, and describe the work procedure manuals. We will work while comparing the situation at the site. For this reason, it takes time to select a necessary part, and an operation error such as a misidentification occurs when a wrong part is selected or compared.
(1-2) Record the work results on a check sheet on a paper medium. Since the work load for digitizing the recorded contents is large, the work results are stored as a check sheet of a paper medium, so that it is difficult to organize and analyze the work results.

Regarding the above-mentioned problems, the following improvement effect is expected by applying the image processing technology using augmented reality to the above-mentioned field work.
(2-1) Just by photographing a work target object using a camera of an image processing apparatus that stores an electronic manual, the work location and procedure related to the work target object are displayed on a display image (camera preview image and Display information (may also be referred to as object icons) in real time in the form of superimposed display. As a result, it is possible to shorten the user's work time and reduce work mistakes.
(2-2) The work result recording work is performed by switching to a work result input screen by, for example, pressing and selecting display information displayed in association with an object to be worked with a finger. . When the work result recording is completed, the work result is already digitized, so that the work result data can be easily organized and revised.
As described above, in the image processing technology using augmented reality, various work efficiencies at the work site are expected.

  "Proposal and Evaluation of Nuclear Power Plant Dismantling Support Method Using Augmented Reality", Transactions of the Virtual Reality Society of Japan, Vol. 2, pp. 289-300, 2008

The display information display operation in the image processing technology using augmented reality based on the recognition of the current marker operates based on the following processing, for example.
(3-1) While the recognition of the marker included in the image is successful, the display information is displayed based on the relative position of the marker with respect to the reference position with reference to the recognition position and orientation of the marker in the image.
(3-2) The display position of the display information changes in real time following changes in the shooting position and direction with respect to the marker generated according to the user's operation.
(3-3) Display information is immediately displayed when the marker moves out of the shooting range due to movement of the shooting position or changes in posture, or when marker recognition fails due to the occurrence of blurring during shooting, etc. Stop.
(3-4) When display information is displayed, the user performs a function assigned to the selection information by selecting the display information by pressing with a finger.

The augmented reality technology is a technical feature in which the item (3-2) is prominent among the items (3-1) to (3-4) described above. Specifically, since the display position of the display information sequentially changes in real time in accordance with the shooting range during imaging, the following processing can be performed only by moving the position of the image processing apparatus without performing a special operation such as button operation. An effect can be obtained.
(4-1) The presence / absence of display information in the shooting range can be confirmed simply by visually recognizing the display while moving the image processing apparatus (ensuring listability of display information).
(4-2) The photographing position can be easily adjusted so that the display position of the object to be worked and the display information becomes an appropriate place and size at the time of work. In addition, when it is difficult to visually recognize display information due to mirror reflection of an object with respect to a work target or reflection on a display, it is possible to easily adjust the display information to be displayed at a position where it can be easily viewed (display information Display range controllability).

  As can be understood from the fact that the display information listability and controllability of the display range of the display information are secured as described above, in image processing using augmented reality, high usability (usability) for the user is achieved. It is in the required state. In ensuring convenience, it is also important to ensure the accuracy of pressing the user's display information. This is because, as will be described later, the user holds the image processing apparatus while extending the finger to the display information so that the user can perform a pressing selection operation by touching one of the fingers with respect to the display information on the display. This is because it has been clarified by the present inventors that the position of the user's finger becomes unstable and the position and posture of the image processing apparatus on which the camera is mounted suddenly moves. An object of the present invention is to provide an image processing apparatus that improves usability for a user.

  An image processing device disclosed in the present invention includes a display unit that displays display information, and an acquisition unit that acquires an image including a marker that defines the display information and a reference position for displaying the display information. Further, the image processing apparatus includes a recognition unit that recognizes display information and a reference position defined by the marker, and a control unit that controls a user operation area for the display information wider than the display area of the display information.

  The objects and advantages of the invention may be realized and attained by means of the elements and combinations in the claims, for example. It should also be understood that both the above general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention as claimed.

  In the image processing apparatus disclosed in this specification, it is possible to improve the pressing accuracy of the display information of the user.

It is a functional block diagram of the image processing apparatus 1 by 1st Embodiment. (A) is a conceptual diagram before the selection operation of the display information by a user is started. (B) is a conceptual diagram after the display information selection operation by the user is started. It is an example of the 1st table containing the movement amount of the reference position in selection operation time. It is a conceptual diagram of the superimposed display of the display information in the movement range of the reference position of the selection operation time. 3 is a flowchart of image processing by the image processing apparatus 1. It is a definition diagram of a rotation vector and a translation vector. It is a functional block diagram of the image processing apparatus 1 by 2nd Embodiment. It is an example of the 2nd table containing the movement amount of the reference | standard position in selection operation time. (A) is a conceptual diagram of display information selection operation start time ts by the user. (B) is a conceptual diagram of the display information selection operation end time tp by the user. (C) is a conceptual diagram of the next frame time tp + 1 of the display information selection operation end time by the user. It is a hardware block diagram of the computer which functions as the image processing apparatus 1 by one Embodiment.

  In the image processing using the augmented reality in which the display position of the display information fluctuates in real time, the following matters have been clarified by the inventors' extensive examination. Since the user performs a pressing selection operation by touching one finger with respect to the display information on the display, the position of the other finger holding the image processing apparatus is stabilized while the finger is extended to the display information. The phenomenon that the position and posture of the image processing apparatus on which the camera is mounted moves suddenly occurs. In this case, since the camera photographing position and direction with respect to the marker change, the display position of the display information changes. As a result, the display position of the display information at the time when the user's finger touches the display is misaligned with the display position at the time when the pressing selection operation is started, so that the user cannot press the display information accurately. It became clear that it occurred. It has also been clarified that this phenomenon occurs even when the display information is pressed while holding the image processing apparatus with both hands. This is because even when the image processing apparatus is held with both hands, the position and orientation of the image processing apparatus change due to the operation of extending the finger and the change of the image processing apparatus accompanying the operation.

  As a comparative example, the present inventors have examined the following method with respect to changes in the display position of display information accompanying changes in the position and orientation of the image processing apparatus. For example, a method is conceivable in which the display information is fixedly displayed at a predetermined position on the display, and is linked to the work target object by an arrow on the moving image. According to this method, even if the position or orientation of the image processing apparatus changes, the display position of the display information does not change, so that the user can press the display information accurately. However, this method has a problem that the display information and the object to be worked are separated on the display and the visibility is deteriorated.

  Further, as another comparative example, by pressing a specific display information displayed on the display, the real-time image display is temporarily switched to the still image display, and then the display information on the still image is changed. It is conceivable that the user presses the button. However, since the pressing operation itself for switching from the real-time image display to the still image display requires pressing down to specific display information in the real-time display information, there is a problem that the position and orientation of the image processing apparatus described above change. Arise. In other words, in an image processing apparatus that uses augmented reality, the display position of display information varies in real time. For this reason, even if the display position of the display information fluctuates in real time, it is necessary to ensure sufficient pressing accuracy for the display information of the user.

  Based on the drawings, an example of an image processing apparatus, an image processing method, and an image processing program according to one embodiment will be described below in consideration of newly discovered items by the above-described diligent verification by the present inventors. This will be described in detail. Note that this embodiment does not limit the disclosed technology.

Example 1
FIG. 1 is a functional block diagram of an image processing apparatus 1 according to the first embodiment. The image processing apparatus 1 includes an acquisition unit 2, a recognition unit 3, a detection unit 4, a calculation unit 5, a control unit 6, and a display unit 7. Note that the image processing apparatus 1 may be configured by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

  The acquisition unit 2 is a hardware circuit based on wired logic, for example. The acquisition unit 2 may be a functional module realized by a computer program executed by the image processing apparatus 1. Furthermore, the acquisition unit 2 includes a communication unit (not shown), and can bidirectionally transmit and receive data to and from various external devices via a communication line. The acquisition unit 2 acquires an image including display information displayed by the display unit 7 described later and a marker that defines a reference position for displaying the display information. Here, the display information is information including work contents associated with an object to be worked, for example. The acquisition unit 2 acquires the image from, for example, an image sensor (not shown) (also referred to as a camera) that is an example of an external device. The imaging device is an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) camera, for example. Further, the image processing apparatus 1 can make the imaging element a part of the components of the image processing apparatus 1. The acquisition unit 2 outputs the acquired image to the recognition unit 3.

  The recognition unit 3 is a hardware circuit based on wired logic, for example. Further, the recognition unit 3 may be a functional module realized by a computer program executed by the image processing apparatus 1. The recognition unit 3 receives an image from the acquisition unit 2, and recognizes display information and a reference position corresponding to a marker included in the image. The recognition unit 3 outputs the recognized display information and the reference position to the control unit 6. The recognition unit 3 may include a storage unit (not shown). For example, display information corresponding to the marker is stored in the storage unit. The storage unit is, for example, a semiconductor memory device such as a flash memory, or a storage device such as an HDD (Hard Disk Drive) or an optical disk. Note that the storage unit is not limited to the above-described type of storage device, and may be a RAM (Random Access Memory) or a ROM (Read Only Memory). The recognition unit 3 outputs the recognized reference position and display information to the calculation unit 5, the control unit 6, and the display unit 7.

  The detection unit 4 is a hardware circuit based on wired logic, for example. The detection unit 4 may be a functional module realized by a computer program executed by the image processing apparatus 1. The detection unit 4 detects the contact of the user's finger with the display unit 7 described later, and detects the detection position on the display unit 7. In addition, the detection part 4 can prescribe | regulate the time which detected the contact as the end time of the user's selection operation | movement with respect to display information. The detection unit 4 detects the selection of the user's display information based on the detection position where the contact is detected and the user's operation area for the display information displayed on the display unit 7. The operation area is, for example, a display area for display information. The detection unit 4 outputs the detected end time of the selection operation to the calculation unit 5.

  The calculation unit 5 is a hardware circuit based on wired logic, for example. The calculation unit 5 may be a functional module that is realized by a computer program executed by the image processing apparatus 1. The calculation unit 5 receives the reference position and the display position from the recognition unit 3. Further, the calculation unit 5 receives the end time of the selection operation from the detection unit 4. The calculation unit 5 calculates start information of a user's selection operation for display information displayed by the display unit 7 to be described later. Here, the start information includes the start time of the selection operation based on the movement amount of the reference position. In other words, the start time is a time when the first display position of the display information satisfies the threshold value not less than the first threshold range and the threshold value not less than the second threshold time and then the threshold value not less than the first threshold range. Furthermore, in other words, the start time satisfies the threshold value not less than the first threshold range after the change amount of the first display position of the display information satisfies the threshold value less than the first threshold range and not less than the second threshold time. It's time. Specifically, when the user has stopped the image processing apparatus 1 according to the marker, the display position of the display information (corresponding to the first display position) is a certain time or more (corresponding to the second threshold time or more). Over an almost stationary state (corresponding to less than the first threshold range). If the details of the first display position are supplementarily described, the display position of the display information at the start time of the selection operation can be rephrased as the first display position. Next, when the user's finger moves toward the display information, the image processing apparatus 1 changes the holding state of the user, so the display position of the display information is within a certain range (corresponding to the first threshold range or more). fluctuate. By using such a processing method, the calculation unit 5 can calculate the start information of the user's selection operation.

  FIG. 2A is a conceptual diagram before the start of the display information selection operation by the user. FIG. 2B is a conceptual diagram after the display information selection operation by the user is started. As shown in FIG. 2A, before the display information selection operation by the user is started, the user holds the image processing apparatus 1 including the display unit 7 with both hands and rests the image processing apparatus 1 according to the marker. It is in a state. For this reason, the display position of display information (corresponding to the first display position) is substantially stationary (corresponding to less than the first threshold range) for a certain time or more (corresponding to the second threshold time or more). The display unit 7 of the image processing apparatus 1 displays display information at the left end, an object to be worked at the center, and a marker at the right end. As shown in FIG. 2B, since the image processing apparatus 1 is held with one hand after the display information selection operation is started, the position of the finger holding the image processing apparatus 1 becomes unstable, and the display position of the display information is displayed. Fluctuates over a certain range (corresponding to the first threshold range or more). In other words, the display position of the display information is different before and after the start of the selection operation.

  The calculation unit 5 in FIG. 1 calculates the selection operation time from the start time included in the start information to the end time of the selection operation. Note that the selection operation time is not simply information on the time length, and may include, for example, the start time and end time of the selection operation. The calculation unit 5 outputs the calculated selection operation time to the control unit 6.

  The control unit 6 is a hardware circuit based on wired logic, for example. The control unit 6 may be a functional module realized by a computer program executed by the image processing apparatus 1. The control unit 6 receives the reference position and display information from the recognition unit 3 and receives the selection operation time from the calculation unit 5. Note that the selection operation time in the first embodiment includes information on the start time and end time of the selection operation. The control unit 6 controls the user operation area for the display information to be wider than the display information display area. Moreover, the control part 6 controls the operation area | region of display information based on start information. The control unit 6 controls the operation area based on the start information and the movement amount of the reference position. Furthermore, the control unit 6 controls the operation area based on the movement amount during the selection operation time. Further, the control unit 6 controls the operation area wider than the display area at the start time (may be referred to as an initial display area). Specifically, as control of the operation area, the control unit 6 causes display information to be superimposed and displayed on a moving range defined by the minimum coordinate and the maximum coordinate of the reference position during the selection operation time, for example. Details of the control process of the control unit 6 will be described below.

  The control unit 6 stores a table storing the movement amount of the reference position from the start time ts to the end time tp (for example, the unit of the movement amount is a pixel) of the selection operation of the user with respect to the display information. Store in a cache or memory (not shown). FIG. 3 is an example of a first table including the amount of movement of the reference position during the selection operation time. The table 30 shown in FIG. 3 stores a reference position for each frame of an image acquired by the acquisition unit 2 where the start time of the user's selection operation is ts and the end time is tp. In the first embodiment, it is assumed that the marker has a rectangular shape that can be expressed by two-dimensional coordinates, and the reference point is expressed by four vertices. Further, the reference point may be an arbitrary point instead of the four vertices. The two-dimensional coordinate system in the table 30 is, for example, a coordinate system in which the upper left corner of the display unit 7 is the origin, the horizontal direction is defined as the x axis, and the vertical direction is defined as the y axis. Time tp + 1 indicates the time of the next frame after the time when the detection unit 4 detects contact.

  In the table 30 of FIG. 3, the minimum coordinate of the x axis is 10 and the maximum coordinate of the y axis is 340. This range corresponds to the movement range defined by the minimum coordinate and the maximum coordinate of the reference position in the selection operation time described above. Note that the movement range may be referred to as the above-described movement locus. For example, as the control of the operation area, the control unit 6 superimposes and displays the display information on the movement range defined by the minimum coordinate and the maximum coordinate of the reference position in the selection operation time. Thereby, the pressing precision with respect to a user's display information improves. FIG. 4 is a conceptual diagram of superimposed display of display information in the movement range of the reference position for the selection operation time. In FIG. 4, the position pressed by the user's fingertip (detected position at time tp) is a position other than the display position at time ts (time tp-1), but the pressed position is a reference in the selection operation time. Since it is within the movement range of the operation area defined by the minimum coordinates and the maximum coordinates of the position, the detection unit 4 detects the selection of the display information by the user, thereby improving the pressing accuracy of the display information of the user.

  Note that the control unit 6 does not necessarily display the display information in a superimposed manner. For example, if the detection position at which the detection unit 4 detects contact is within the movement range defined by the control unit 6, the detection unit 4 detects that the selection of the display information has been made, thereby detecting the user. The pressing accuracy for the display information is improved. In other words, the control unit 6 controls the operation region by, for example, considering the region of the movement range defined by the minimum coordinate and the maximum coordinate of the reference position during the selection operation time as the operation range, and then detecting the detection unit 4. By detecting the selection of display information, the effective operation range is expanded, and the accuracy of pressing the display information by the user is improved. For convenience of explanation, various information for the control unit 6 to control the display position of the display information will be referred to as control information. The control unit 6 outputs control information to the display unit 7.

Further, the control unit 6 may calculate a time change of the reference position in each frame using the reference positions at two different times, and may control the display information based on the calculated movement range of the reference position. good. The time change of the reference position can be calculated using, for example, the following equation. In the following equation, x and y are the coordinates of the reference point, and t is the time of an arbitrary frame.
(Equation 1)
x = 5t + 35
y = -15t + 165
In addition, when the time change of the reference position is a curve, the control unit 6 may calculate the movement locus using an approximate curve equation. In addition, the control unit 6 may calculate the time change of the reference position in each frame based on coordinates taking into account a rotation vector and a translation vector, which will be described later.

  The display unit 7 in FIG. 1 is a display device such as a liquid crystal display or an organic electroluminescence display. The display unit 7 receives display information and a reference position from the recognition unit 3, and receives display information control information from the control unit 6. The display unit 7 displays the display information based on the control information when the control information is received from the control unit 6, and displays the display information based on the reference position when the control information is not received from the control unit 6. Is displayed.

  FIG. 5 is a flowchart of image processing by the image processing apparatus 1. The acquisition unit 2 acquires an image including display information displayed by the display unit 7 and a marker that defines a reference position for displaying the display information (step S501). The acquisition unit 2 outputs the acquired image to the recognition unit 3. The recognition unit 3 receives the image from the acquisition unit 2, and recognizes the display information and the reference position defined by the marker included in the image (step S502). The recognition unit 3 outputs the recognized display information and the reference position to the control unit 6.

  The detection unit 4 detects contact of the user's finger with the display unit 7 and detects a detection position on the display unit 7 (step S503). In addition, the detection part 4 can prescribe | regulate the time which detected the contact as the end time of the user's selection operation | movement with respect to display information. The detection unit 4 detects the selection of the user's display information based on the detection position where the contact is detected and the user's operation area for the display information displayed on the display unit 7. The operation area is, for example, a display area for display information. The detection unit 4 outputs the detected end time of the selection operation to the calculation unit 5.

  The calculation unit 5 receives the reference position and the display position from the recognition unit 3. Further, the calculation unit 5 receives the end time of the selection operation from the detection unit 4. The calculation unit 5 calculates start information of the user's selection operation for the display information displayed by the display unit 7 (step S504). Here, the start information includes the start time of the selection operation based on the movement amount of the reference position. In other words, the start time is a time when the first display position of the display information satisfies the threshold value not less than the first threshold range and the threshold value not less than the second threshold time and then the threshold value not less than the first threshold range.

  The calculation unit 5 calculates the selection operation time from the start time included in the start information to the end time of the selection operation (step S505). Note that the selection operation time is not simply information on the time length, and may include, for example, the start time and end time of the selection operation. The calculation unit 5 outputs the calculated selection operation time to the control unit 6. The calculation unit 5 outputs the calculated selection operation time to the control unit 6.

  The control unit 6 receives the reference position and display information from the recognition unit 3 and receives the selection operation time from the calculation unit 5. The selection operation time includes information on the start time and end time of the selection operation. The control unit 6 controls the display area of the display information based on the selected time operation (step S506). For example, in step S506, the control unit 6 sets the user's operation area for the display information more than the display area of the display information. Widely control. Moreover, the control part 6 controls the operation area | region of display information based on start information. The control unit 6 controls the operation area based on the start information and the amount of movement of the reference position. Furthermore, the control unit 6 controls the operation area based on the movement amount during the selection operation time. Specifically, as control of the operation area, the control unit 6 causes display information to be superimposed and displayed on a moving range defined by the minimum coordinate and the maximum coordinate of the reference position during the selection operation time, for example.

In step S506, the control unit 6 does not necessarily display the display information in a superimposed manner. For example, if the detection position at which the detection unit 4 detects contact is within the movement range defined by the control unit 6, the detection unit 4 detects that the selection of the display information has been made, thereby detecting the user. The pressing accuracy for the display information is improved. In other words, the control unit 6 controls the operation region by, for example, considering the movement range region itself defined by the minimum coordinate and the maximum coordinate of the reference position in the selection operation time as the operation range, When 4 detects the selection of display information, the effective operation range is expanded, and the pressing accuracy with respect to the display information of the user is improved.
For convenience of explanation, various information for the control unit 6 to control the display position of the display information will be referred to as control information. The control unit 6 outputs control information to the display unit 7.

  The display unit 7 receives display information and a reference position from the recognition unit 3, and receives display information control information from the control unit 6. The display unit 7 displays the display information (Step S507). For example, if the display unit 7 has received control information from the control unit 6 in step S507, the display unit 7 displays the display information based on the control information. If the control unit 6 has not received control information, the display unit 7 By displaying the display information based on the position, the image processing apparatus 1 ends the image processing shown in the flowchart of FIG. Note that when the acquisition unit 2 has received an image at the end of the process in step S507, the image processing apparatus may repeatedly execute the processes in steps S501 to S507.

  In the image processing apparatus 1 according to the first embodiment, usability for the user can be improved. Further, in the image processing apparatus 1 according to the first embodiment, it is possible to improve the pressing accuracy with respect to the display information of the user. For example, even when the display position of the display information moves with the movement of the image processing apparatus 1, a display that is temporally close to the display position at the time when the user has aimed for the operation is displayed or selected. As a result, it is possible to improve the pressing accuracy for the display information of the user.

(Example 2)
In the second embodiment, a method for calculating the selection operation time using the rotation vector and the translation vector by the calculation unit 5 will be disclosed. Since the functional block diagram of the image processing apparatus 1 including the calculation unit 5 in the second embodiment is the same as that in the first embodiment, detailed description thereof is omitted.

  FIG. 6 is a definition diagram of the rotation vector and the translation vector. As shown in FIG. 6, a translation vector Tz is defined perpendicularly from the origin 0 at an arbitrary position to the display unit 7 of the image processing apparatus 1, and translation vectors Tx and Ty corresponding to the translation vector Tz are respectively defined. . Further, rotation vectors Rx, Ry, and Rz are respectively defined with respect to the rotation directions of the translation vectors Tx, Ty, and Tz. For example, the calculation unit 5 calculates a rotation vector R = (Rx, Ry, Rz) and a translation vector T = (Tx, Ty, Tz) from a reference point represented by four vertices as shown in the table 30 of FIG. It calculates using a well-known method.

The calculation unit 5 calculates the translation vector Tn = (Txn, Tyn, Tzn) at the time tn (for example, the time ts + 2 in the table 30 in FIG. 3) and the time tn−1 (for example, the table in FIG. 3). From the translation vector Tn−1 = (Txn−1, Tyn−1, Tzn−1) at time 30 (ts + 1)), a translation vector amount ΔTn representing the movement amount from time tn−1 to time tn is expressed by the following equation: Calculate using.
(Equation 2)

Similarly to the translation vector, the calculation unit 5 calculates a rotation vector amount ΔRn that represents a movement amount from time tn−1 to time tn using the following equation.
(Equation 3)

  When the translation vector amount ΔTn is less than the predetermined value ΔTα and the rotation vector amount ΔRn is less than the predetermined value ΔRα, the calculation unit 5 determines that the image processing apparatus 1 is in a stationary state. Next, the calculation unit 5 calculates the duration t1 of the stationary state. When the duration t1 is less than the predetermined time tβ, the calculation unit 5 resets the duration to 0 and then repeatedly calculates the translation vector amount ΔTn and the rotation vector amount ΔRn. When the duration t1 is equal to or longer than the predetermined time tβ, the calculation unit 5 executes the following process.

The calculation unit 5 compares the translation vector amount ΔTn with the predetermined value ΔTα and also compares the rotation vector amount ΔRn with the predetermined value ΔRα. When ΔTn <= ΔTα and ΔRn <= ΔRα, the calculation unit 5 determines that the image processing apparatus 1 is still in a stationary state. The calculation unit 5
When ΔTn> ΔTα or ΔRn> ΔRα is changed, it is determined that the position of the image processing apparatus 1 has changed, and the time when the above-described condition is satisfied can be defined as the start time ts of the selection operation.

  In the image processing apparatus 1 according to the second embodiment, the selection operation is started with high accuracy by using the change amounts of the six measurement values of the translation vector in the three-axis direction and the rotation vector in the three-axis direction and the plurality of threshold determinations. It can be defined as time ts. For this reason, in the image processing apparatus 1 according to the second embodiment, the usability for the user can be improved. Further, in the image processing apparatus 1 according to the second embodiment, it is possible to improve the pressing accuracy with respect to the display information of the user.

(Example 3)
When the user uses the image processing apparatus 1, the recognition unit 3 may not be able to recognize the marker due to occurrence of blurring or the like. When the recognition unit 3 does not recognize the marker, the display information may not be displayed on the display unit 7 due to the disappearance of the reference position. In this case, the user cannot press the display information with a finger or the like, and the availability may be reduced. In the image processing apparatus 1 according to the third embodiment, the image processing apparatus 1 that can prevent the availability from being lowered even when the marker cannot be recognized will be described.

  FIG. 7 is a functional block diagram of the image processing apparatus 1 according to the second embodiment. The image processing apparatus 1 includes an acquisition unit 2, a recognition unit 3, a detection unit 4, a calculation unit 5, a control unit 6, a display unit 7, and an estimation unit 8. Note that the image processing apparatus 1 may be configured by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Moreover, since each function other than the estimation part 8 has a function similar to each function shown in FIG. 1, detailed description is abbreviate | omitted.

  The estimation unit 8 is a hardware circuit based on wired logic, for example. Further, the estimation unit 8 may be a functional module realized by a computer program executed by the image processing apparatus 1. When the calculating unit 5 has not received the reference position from the recognizing unit 3, the estimating unit 8 performs the following estimation process.

  FIG. 8 is an example of a second table including the amount of movement of the reference position during the selection operation time. Since various specifications such as the coordinate system in the table 80 are the same as those in the table 30 of FIG. 3, detailed description thereof is omitted. The table 80 shown in FIG. 8 stores a reference position for each frame of an image acquired by the acquisition unit 2 where the start time of the user's selection operation is ts and the end time is tp. The table 80 stores marker recognition results. The marker recognition result may be stored based on, for example, whether the recognition unit 3 has recognized the marker. The table 80 indicates that marker recognition has succeeded at time ts to time tm−1 and marker recognition has failed at time tm to time tp + 1. The estimation unit 8 uses the change in the reference position from time ts to tm−1 before the marker recognition failure time, and uses the movement amount (movement trajectory) of the reference position from the marker recognition failure time tm to the time tp at which contact is detected. May be referred to).

The estimation unit 8 can use various known methods for estimating the amount of movement of the reference position from the marker recognition failure time tm to the time tp at which contact is detected. For example, a linear approximation method based on a time change that can be calculated from a reference position at time ts to time tm−1, “Computer Vision, David A. Forsyth, Jean Ponce (Author), Takeshi Ohkita (translation) ), Kyoritsu Shuppan (2007) ", or" Particle filter and its implementation (tutorial), IPSJ Research Report, CVIM, 2007 (1), 161-168, 2007-01-11 " Various filter processes such as a particle filter disclosed in the above can be used. In the third embodiment, linear approximation based on time change will be described as an example. In the table 80, for example, the time change of the reference position from time tm-3 to time tm-1 can be expressed by the following equation, for example. In the following equation, x and y are the coordinates of the reference point, and t is the time of an arbitrary frame.
(Equation 4)
x = 5t + 35
y = -15t + 165

  The estimation unit 8 can estimate the amount of movement from time tm to time tp using the above (Equation 4). The estimated coordinates of the reference position are the coordinates shown in the table 80. The control unit 6 in FIG. 7 refers to the table 80 and appropriately executes the control process disclosed in the first embodiment or the second embodiment.

According to the image processing apparatus 1 according to the third embodiment, the position of the display information can be estimated and displayed even when the display information is not accurately displayed during the user's selection operation. For this reason, in the image processing apparatus 1 according to the third embodiment, the usability for the user can be improved. Further, according to the image processing apparatus 1 according to the third embodiment, it is possible to improve the pressing accuracy with respect to the display information of the user.

Example 4
The control unit 6 in the image processing apparatus 1 disclosed in the first to third embodiments selects the second display position of the display information in the next frame after the end time of the selection operation time as the control of the operation area of the display information. The display information may be displayed by switching to the first display position of the display information of the time start time. In addition, if a 2nd display position is demonstrated supplementarily, the display position of the display information in the next frame of the end time of selection operation time will be made into a 2nd display position. In particular, in the fourth embodiment, it is more preferable to perform the processing when the detection position where the detection unit 4 detects contact is outside the display area. Since the functional block diagram of the image processing apparatus 1 including the control unit 6 in the fourth embodiment is the same as that in the first to third embodiments, detailed description thereof is omitted.

  FIG. 9A is a conceptual diagram of the display information selection operation start time ts by the user. FIG. 9B is a conceptual diagram of the display information selection operation end time tp by the user. FIG. 9C is a conceptual diagram of the next frame time tp + 1 of the display information selection operation end time by the user. From the selection operation start time ts to the selection operation end time tp in FIG. 9A, the display position of the display information is moved with the movement of the image processing apparatus 1. In this case, the control unit 6 stores the reference position of the selection operation start time ts at the selection operation start time ts. Next, as illustrated in FIG. 9C, the control unit 6 determines, based on the reference position of the stored selection operation start time ts, at the time tp + 1 that is the next frame of the display information selection operation end time tp. Control display information to be displayed. In other words, the display position of the display information at time ts and time tp + 1 is the same position. The process is more preferably performed when the detection position at which the detection unit 4 detects contact is outside the display area and within the operation area range. In the image processing apparatus 1 according to the fourth embodiment, usability (visibility) for the user can be improved.

  Further, in FIGS. 9A to 9C, the control unit 6 determines that the detection position of the display information selection operation end time tp by the user is the reference position of the selection operation start time ts and the selection operation end time tp. It may be controlled to determine which of the reference positions is close and to display the display information based on the reference position of the close position. In other words, the display position of the display information at time tp + 1 is the display position at either time ts or time tp. By this operation, a display that is close in time to the display position at the time when the user has aimed for the operation is displayed or selected, so that the usability (visibility) for the user can be improved.

  Further, in FIG. 9A to FIG. 9C, the control unit 6 does not change the table 30 in FIG. 3 or the table 30 in FIG. 3 even when the detection position of the display information selection operation end time tp by the user is outside the display area. The display position of the display information at time tp + 1 may be superimposed on the detection position at the selection operation end time tp as long as it is within the movement range defined by the control unit 6 based on the table 80 of FIG. With this operation, usability (visibility) for the user can be improved.

(Example 5)
Since the functional block diagram of the image processing apparatus 1 including the control unit 6 in the fifth embodiment is the same as that in the first to fourth embodiments, detailed description thereof is omitted. 9A to 9C, the control unit 6 can also fix the display position of the display information from the selection operation start time ts to the selection operation end time tp. In the operation, since the display position itself at the time when the user sets the target of the operation continues to be displayed, it is possible to improve the pressing accuracy with respect to the display information of the user. In the image processing apparatus 1 according to the fifth embodiment, usability for the user can be improved. Further, according to the image processing apparatus 1 according to the fifth embodiment, it is possible to improve the pressing accuracy with respect to the display information of the user.

  In each embodiment described above, the image processing apparatus 1 may further include a microphone that acquires a user's voice signal. The microphone may be connected to the calculation unit 5, for example. Further, as the audio signal, for example, a signal including a specific word that can be recognized can be used. The calculation unit 5 may calculate an audio signal as the above start information.

  The image processing apparatus 1 can further include an acceleration sensor, an angular velocity sensor, or a contact switch that acquires a user's operation signal. The acceleration sensor, the angular velocity sensor, or the contact switch may be connected to the calculation unit 5, for example. The operation signal can be an operation amount set with a predetermined threshold or an electrical contact signal by a switch. The calculation unit 5 may calculate the operation signal as the above-described start information.

  The image processing apparatus 1 can further include a camera that captures a selection operation image including a selection operation. The camera may be connected to the calculation unit 5, for example. Moreover, a camera images a user's finger, for example. The calculation unit 5 can calculate start information based on a processing result such as template matching for the selected operation image.

  Further, the control unit 6 of the image processing apparatus 1 may simply enlarge and display the display area of the display information based on the start information. This process can also improve the pressing accuracy by the user. Furthermore, when the recognition unit 3 recognizes the marker, the control unit 6 may enlarge and display the display area of the display information. Specifically, before the display unit 7 displays the display information, the control unit 6 of the image processing apparatus 1 may control the operation region to be wider than the display information display region recognized by the recognition unit 3. . For example, when the recognition unit 3 recognizes a marker, the control unit 6 may simply enlarge and display the display area of the display information associated with the marker. This process can also improve the pressing accuracy by the user. Further, when the display form is such that the user presses a partial area of the display information, only the partial area may be displayed in an enlarged manner. This process can also improve the pressing accuracy by the user. The display information includes incidental information associated with the display information, and the control unit 6 can display the incidental information when the detection unit 4 detects selection of the display information.

(Example 6)
FIG. 10 is a hardware configuration diagram of a computer that functions as the image processing apparatus 1 according to an embodiment. As shown in FIG. 10, the image processing apparatus 1 includes a computer 100 and an input / output device (peripheral device) connected to the computer 100.

  The computer 100 is entirely controlled by a processor 101. The processor 101 is connected to a RAM (Random Access Memory) 102 and a plurality of peripheral devices via a bus 109. The processor 101 may be a multiprocessor. In addition, the processor 101 is, for example, a CPU, an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or a PLD (Programmable Logic D). Further, the processor 101 may be a combination of two or more elements of CPU, MPU, DSP, ASIC, and PLD.

  The RAM 102 is used as a main storage device of the computer 100. The RAM 102 temporarily stores at least a part of an OS (Operating System) program and application programs to be executed by the processor 101. The RAM 102 stores various data necessary for processing by the processor 101.

  Peripheral devices connected to the bus 109 include an HDD (Hard Disk Drive) 103, a graphic processing device 104, an input interface 105, an optical drive device 106, a device connection interface 107, and a network interface 108.

  The HDD 103 magnetically writes and reads data to and from the built-in disk. The HDD 103 is used as an auxiliary storage device of the computer 100, for example. The HDD 103 stores an OS program, application programs, and various data. Note that a semiconductor storage device such as a flash memory can be used as the auxiliary storage device.

  A monitor 110 is connected to the graphic processing device 104. The graphic processing device 104 displays various images on the screen of the monitor 110 in accordance with instructions from the processor 101. Examples of the monitor 110 include a display device using a cathode ray tube (CRT) and a liquid crystal display device.

  A keyboard 111 and a mouse 112 are connected to the input interface 105. The input interface 105 transmits signals sent from the keyboard 111 and the mouse 112 to the processor 101. Note that the mouse 112 is an example of a pointing device, and other pointing devices can also be used. Examples of other pointing devices include a touch panel, a tablet, a touch pad, and a trackball.

  The optical drive device 106 reads data recorded on the optical disk 113 using laser light or the like. The optical disk 113 is a portable recording medium on which data is recorded so that it can be read by reflection of light. Examples of the optical disc 113 include a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc Read Only Memory), and a CD-R (Recordable) / RW (ReWriteable). A program stored in the optical disk 113 serving as a portable recording medium is installed in the image processing apparatus 1 via the optical drive device 106. The installed predetermined program can be executed by the image processing apparatus 1.

  The device connection interface 107 is a communication interface for connecting peripheral devices to the computer 100. For example, a memory device 114 or a memory reader / writer 115 can be connected to the device connection interface 107. The memory device 114 is a recording medium equipped with a communication function with the device connection interface 107. The memory reader / writer 115 is a device that writes data to the memory card 116 or reads data from the memory card 116. The memory card 116 is a card type recording medium.

  The network interface 108 is connected to the network 11. The network interface 108 transmits and receives data to and from other computers or communication devices via the network 11.

  The computer 100 implements the above-described image processing function by executing a program recorded on a computer-readable recording medium, for example. A program describing the processing contents to be executed by the computer 100 can be recorded in various recording media. The program can be composed of one or a plurality of functional modules. For example, a program can be comprised from the functional module which implement | achieved the process of the acquisition part 2, the recognition part 3, the detection part 4, the calculation part 5, the control part 6, and the estimation part 8 of FIG. 1 or FIG. Note that a program to be executed by the computer 100 can be stored in the HDD 103. The processor 101 loads at least a part of the program in the HDD 103 into the RAM 102 and executes the program. A program to be executed by the computer 100 can also be recorded on a portable recording medium such as the optical disc 113, the memory device 114, and the memory card 116. The program stored in the portable recording medium becomes executable after being installed in the HDD 103 under the control of the processor 101, for example. The processor 101 can also read and execute a program directly from a portable recording medium.

  In addition, each component of each illustrated apparatus does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. The various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer or a workstation.

The following supplementary notes are further disclosed with respect to the embodiment described above.
(Appendix 1)
A display unit for displaying display information;
An acquisition unit that acquires an image including a marker that defines a reference position for displaying the display information and the display information;
A recognition unit for recognizing the display information and the reference position defined by the marker;
A control unit for controlling a user operation area for the display information wider than a display area of the display information;
An image processing apparatus comprising:
(Appendix 2)
A calculation unit that calculates start information of a user's selection operation for the display information;
The image processing apparatus according to claim 1, wherein the control unit controls an operation area of the display information based on the start information.
(Appendix 3)
The image processing apparatus according to claim 2, wherein the control unit controls the operation area based on the start information and a movement amount of the reference position in the image.
(Appendix 4)
The start information includes a start time of the selection operation based on the movement amount,
The calculation unit further calculates a selection operation time from the start time to an end time of the selection operation,
The image processing apparatus according to claim 3, wherein the control unit controls the operation area based on the movement amount in the selection operation time.
(Appendix 5)
The image processing apparatus according to claim 4, wherein the control unit controls the operation area to be wider than the display area at the start time.
(Appendix 6)
A detection unit for detecting contact with the display unit;
The said detection part detects the selection to the said display information of the said user based on the detection position and the said operation area | region which detected the said contact, The appendix 1 thru | or any one of the appendix 5 characterized by the above-mentioned Image processing device.
(Appendix 7)
The start time is a time when the first display position of the display information satisfies a threshold that is less than a first threshold range and continues for a second threshold time or more and then satisfies a threshold that is equal to or greater than the first threshold range.
The image processing apparatus according to any one of appendix 4 to appendix 6, wherein the end time is a time when the detection unit detects the contact.
(Appendix 8)
The control unit causes the display image to be superimposed and displayed on a moving range defined by the minimum coordinate and the maximum coordinate of the reference position in the selection operation time as control of the operation area. The image processing device according to any one of appendix 7.
(Appendix 9)
The control unit controls the operation area by considering a movement range defined by a minimum coordinate and a maximum coordinate of the reference position in the selection operation time as the operation area. Or the image processing device according to any one of Appendix 7.
(Appendix 10)
In the case where the detection position is other than the display area, the control unit controls the operation area as the second display position of the display information in the next frame of the end time, the second display position of the display image at the start time. The image processing apparatus according to any one of appendices 7 to 8, wherein the display information is displayed by switching to one display position.
(Appendix 11)
And an estimation unit that estimates the amount of movement during the selection operation time using a predetermined approximation process or a predetermined filter process when the recognition unit does not recognize a marker. The image processing apparatus according to any one of 9.
(Appendix 12)
The image processing apparatus according to any one of appendices 1 to 11, wherein the operation area is a display area for the display information.
(Appendix 13)
A microphone for acquiring the user's voice signal;
The image processing apparatus according to appendix 2, wherein the calculation unit calculates the audio signal as the start information.
(Appendix 14)
An acceleration sensor, an angular velocity sensor, or a contact switch for acquiring the user's operation signal;
The image processing apparatus according to claim 2, wherein the calculation unit calculates the operation signal as the start information.
(Appendix 15)
A camera that captures a selection operation image including the selection operation;
The image processing apparatus according to appendix 4, wherein the calculation unit calculates the start information based on a processing result of the selected action image.
(Appendix 16)
The image processing apparatus according to claim 2, wherein the control unit enlarges and displays a display area of the display information based on the start information.
(Appendix 17)
The image processing apparatus according to claim 4, wherein the control unit performs control to fix the operation area to the display area at the start time.
(Appendix 18)
The control unit determines whether the detection position of the end time is closer to the reference position of the start time or the reference position of the end time, and the display information is based on the reference position of the close position The image processing apparatus according to appendix 6, wherein:
(Appendix 19)
In the case where the detection position is outside the display area and in the operation area, the control unit changes the second display position of the display information in the next frame of the end time to the display position of the display image at the end time. The image processing apparatus according to appendix 6, wherein the display information is displayed by switching.
(Appendix 20)
The display information includes incidental information associated with the display information,
The image processing apparatus according to appendix 6, wherein the control unit displays the supplementary information when the detection unit detects selection of the display information.
(Appendix 21)
Display information,
Obtaining an image including a marker that defines a reference position for displaying the display information and the display information;
Recognizing the display information defined by the marker and the reference position;
Controlling the user's operation area for the display information wider than the display area of the display information;
An image processing method comprising:
(Appendix 22)
On the computer,
Display information,
Obtaining an image including a marker that defines a reference position for displaying the display information and the display information;
Recognizing the display information defined by the marker and the reference position;
Controlling the user's operation area for the display information wider than the display area of the display information;
An image processing program for executing

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Acquisition part 3 Recognition part 4 Detection part 5 Calculation part 6 Control part 7 Display part 8 Estimation part

Claims (14)

A display unit for displaying display information;
An acquisition unit that acquires an image including the display information and a marker that defines a reference position for displaying the display information;
A recognition unit for recognizing the display information and the reference position defined by the marker;
A calculation unit that calculates start information of a user's selection operation with respect to the display information based on a movement amount of the reference position ;
Based on the start information, a control unit that controls a user operation area for the display information wider than the display area of the display information;
An image processing apparatus comprising: A detection unit for detecting a selection operation on the display information of the user by detecting contact with the display unit;
The image processing apparatus according to claim 1, wherein the control unit controls an operation area of the user for the display information after the detection unit detects a selection operation for the display information of the user.   The image processing apparatus according to claim 2, wherein the control unit controls the operation region based on the start information and an amount of movement of the reference position in the image. The start information includes a start time of the selection operation based on the movement amount,
The calculation unit further calculates a selection operation time from the start time to an end time of the selection operation,
The image processing apparatus according to claim 3, wherein the control unit controls the operation area based on the movement amount in the selection operation time.   The image processing apparatus according to claim 4, wherein the control unit controls the operation area to be wider than the display area at the start time.   The image processing apparatus according to claim 4, wherein the detection unit detects a selection operation for the display information of the user based on a detection position where the contact is detected and the operation region. The start time is a time when the amount of change in the first display position of the display information is less than the first threshold range, and after the second threshold time or more has elapsed, becomes the first threshold range or more.
The image processing apparatus according to claim 6, wherein the end time is a time when the detection unit detects the contact.   5. The control unit according to claim 4, wherein the control information is displayed by superimposing the display information on a movement range defined by a minimum coordinate and a maximum coordinate of the reference position in the selection operation time as control of the operation area. The image processing apparatus according to claim 7.   2. The control unit according to claim 1, wherein as the operation area is controlled, a movement range defined by a minimum coordinate and a maximum coordinate of the reference position in the selection operation time is regarded as the operation area. The image processing apparatus according to any one of claims 4 to 7.   In the case where the detection position is other than the display area, the control unit determines the second display position of the display information in the next frame of the end time as the control of the operation area, the second display position of the display information at the start time. The image processing apparatus according to claim 6, wherein the display information is displayed by switching to one display position.   5. The apparatus according to claim 4, further comprising: an estimation unit configured to estimate the movement amount during the selection operation time using a predetermined approximation process or a predetermined filter process when the recognition unit does not recognize a marker. The image processing apparatus according to claim 9.   The image processing apparatus according to claim 1, wherein the operation area is a display area for the display information. Display information,
Obtaining an image including the display information and a marker defining a reference position for displaying the display information;
Recognizing the display information defined by the marker and the reference position;
Based on the amount of movement of the reference position, calculating start information of the user's selection operation for the display information,
Based on the start information, controlling a user operation area for the display information wider than the display area of the display information,
An image processing method comprising: On the computer,
Display information,
Obtaining an image including the display information and a marker defining a reference position for displaying the display information;
Recognizing the display information defined by the marker and the reference position;
Based on the amount of movement of the reference position, calculating start information of the user's selection operation for the display information,
Based on the start information, controlling a user operation area for the display information wider than the display area of the display information,
An image processing program for executing

Images