WO2011004612A1 - マーカ表示制御装置、集積回路、及び、マーカ表示制御方法 - Google Patents
マーカ表示制御装置、集積回路、及び、マーカ表示制御方法 Download PDFInfo
- Publication number
- WO2011004612A1 WO2011004612A1 PCT/JP2010/004476 JP2010004476W WO2011004612A1 WO 2011004612 A1 WO2011004612 A1 WO 2011004612A1 JP 2010004476 W JP2010004476 W JP 2010004476W WO 2011004612 A1 WO2011004612 A1 WO 2011004612A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- marker
- display
- display device
- unit
- information
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
- H04N21/4318—Generation of visual interfaces for content selection or interaction; Content or additional data rendering by altering the content in the rendering process, e.g. blanking, blurring or masking an image region
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/443—OS processes, e.g. booting an STB, implementing a Java virtual machine in an STB or power management in an STB
- H04N21/4438—Window management, e.g. event handling following interaction with the user interface
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
- H04N21/4722—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for requesting additional data associated with the content
- H04N21/4725—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for requesting additional data associated with the content using interactive regions of the image, e.g. hot spots
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/858—Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot
- H04N21/8583—Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot by creating hot-spots
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2320/00—Control of display operating conditions
- G09G2320/08—Arrangements within a display terminal for setting, manually or automatically, display parameters of the display terminal
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2340/00—Aspects of display data processing
- G09G2340/04—Changes in size, position or resolution of an image
- G09G2340/0407—Resolution change, inclusive of the use of different resolutions for different screen areas
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2340/00—Aspects of display data processing
- G09G2340/04—Changes in size, position or resolution of an image
- G09G2340/0464—Positioning
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2340/00—Aspects of display data processing
- G09G2340/12—Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2370/00—Aspects of data communication
- G09G2370/04—Exchange of auxiliary data, i.e. other than image data, between monitor and graphics controller
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
- G09G5/003—Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto
Definitions
- the present invention relates to a marker display control device that performs control to display a marker for recognizing a display device on a display screen (display area) of the display device, and in particular, along with the original display contents provided to the user by the display device.
- the present invention relates to a marker display control device that performs control to display a marker.
- the marker indicates identification information by shape or color, and can also be called an identification mark.
- a target object with the marker attached thereto is acquired and identification information of the marker is acquired, thereby recognizing the target object or recognizing the position of the target object in the captured image.
- Patent Document 1 describes a technology that uses an information processing terminal equipped with a camera as a remote controller. Then, the identification code (marker) attached to the electronic device is photographed by the information processing terminal, the electronic device to be operated among the plurality of electronic devices is recognized, and the remote operation is performed. This eliminates the need for the user to manually switch the electronic device to be operated. However, it is necessary to prepare a plurality of electronic devices with stickers on which markers are printed and attach them to each electronic device.
- Patent Document 2 a marker is displayed on the display screen of the mobile terminal, and the marker is photographed, and the type of the image associated with the marker, the position of the marker in the photographed image, and the like.
- the technology to recognize is described. Thereby, a plurality of types of markers can be freely displayed on the display screen of the portable terminal without preparing a large number of stickers on which markers corresponding to a plurality of types of images are printed.
- a display device such as a television as an operation target
- a marker together with the display contents inherent to the display device such as an operation menu and a moving image on a display screen (display area) of the display device.
- it is for photographing the marker displayed on the display device and identifying the marker to recognize the type, position, etc. of the display device and facilitate remote operation.
- simply making the marker inconspicuous may make it difficult to detect the marker.
- the shooting distance is long, and detection may be difficult if the marker display size is small.
- shooting of the markers may be hindered by the shielding object, and as a result, it may be difficult to detect the markers. .
- an object of the present invention is to provide a marker display control device that can easily detect a marker and reduce the difficulty in viewing the original display content of the display device.
- a marker display control device is a control device that is connected to a display device by communication means and controls the display device to display a recognition marker.
- a marker detection unit that detects the recognition marker from an image obtained by capturing the content, and a marker that instructs the display device that displays the marker to change the display mode of the marker based on the marker detection result
- a display mode changing unit that changes the display mode of the marker based on the marker detection result
- the marker is displayed in a large size in the display area of the display device before the marker is detected, and the marker display size is reduced after the marker is detected. While being easy, it is possible to reduce the difficulty in viewing the display content inherent in the display device.
- the marker is displayed in a small size in the display area of the display device in a normal state to reduce the difficulty in viewing the display content.
- the marker display size is increased. Thus, the marker can be easily detected.
- the marker display control device of the present invention it is possible to reduce the difficulty in viewing the display content inherent to the display device while facilitating detection of the marker.
- FIG. 3 is a diagram illustrating an appearance configuration and an arrangement example of a small information terminal 1 and a television 2 in the first embodiment.
- 3 is a flowchart showing a display device control process of the small information terminal 1 in the first embodiment.
- 3 is a flowchart showing processing of the television 2 in the first embodiment.
- 3 is a flowchart of marker detection processing in the first embodiment.
- 5 is a flowchart of marker size adjustment processing in the first embodiment.
- 6 is a diagram illustrating the contents of a marker information holding unit 133 during initialization processing in Embodiment 1.
- FIG. 3 is a diagram illustrating an appearance configuration and an arrangement example of a small information terminal 1 and a television 2 in the first embodiment.
- 3 is a flowchart showing a display device control process of the small information terminal 1 in the first embodiment.
- 3 is a flowchart showing processing of the television 2 in the first embodiment.
- 3 is a flowchart of marker detection processing in the first embodiment.
- 5 is a flow
- FIG. 6 is a diagram showing the contents of a marker information holding unit 133 when a marker is detected in Embodiment 1.
- FIG. 6 is a diagram illustrating the contents of a marker information holding unit 133 after adjusting the marker size according to Embodiment 1.
- FIG. 4 is a diagram illustrating a color name of a reference color and a reference luminance value of each color component in the first embodiment.
- 3 is a diagram illustrating an appearance configuration and an arrangement example of a small information terminal 1 after adjusting a marker size and a television 2 in Embodiment 1.
- FIG. 10 is a flowchart showing a non-use detection process in the first modification of the first embodiment.
- FIG. 6 is a block diagram illustrating configurations of a small information terminal 500 and a television 2 in Embodiment 2. It is a figure which shows the content of the marker information holding
- FIG. 10 is a flowchart showing marker size adjustment processing 2 in the second embodiment. 10 is a flowchart showing a non-detectable marker process in the second embodiment. It is a block diagram which shows the structure of the small information terminal 600 in Embodiment 3, television 2, etc.
- FIG. 6 is a diagram showing a small information terminal 600 and a television 2 in Embodiment 3.
- FIG. It is a figure which shows the example of arrangement
- FIG. It is a figure which shows the content of the marker information holding part 133 in Embodiment 3.
- FIG. 10 is a flowchart showing a non-detectable marker process 2 in the third embodiment. 10 is a flowchart showing a shielding state detection process in the third embodiment. 10 is a flowchart showing marker movement processing in the third embodiment. 10 is a flowchart showing movement amount calculation processing in the third embodiment. It is a figure which shows the content of the marker information holding part 133 in Embodiment 3.
- FIG. 10 is a flowchart showing a non-detectable marker process 2 in the third embodiment.
- 10 is a flowchart showing a shielding state detection process in the third embodiment.
- 10 is a flowchart showing marker movement processing in the third
- FIG. It is a figure which shows the example of arrangement
- FIG. It is a figure which shows the moving direction based on the marker display location and random number in the modification of Embodiment 3.
- FIG. It is a figure which shows the content of the marker information holding
- FIG. It is a figure which shows the example of a two-dimensional marker. It is a block diagram which shows the structure of the operating device 700 in another embodiment different from the above, the display apparatus 702, etc.
- a marker display control device is a control device that is connected to a display device by a communication unit and performs control to display a recognition marker on the display device, and the display content of the display device
- a marker detection unit for detecting the marker for recognition from an image obtained by photographing the marker, and a marker display for instructing a display device displaying the marker to change the display mode of the marker based on the marker detection result
- an aspect change unit for detecting the marker for recognition from an image obtained by photographing the marker, and a marker display for instructing a display device displaying the marker to change the display mode of the marker based on the marker detection result.
- the marker display control device of this aspect is mounted on, for example, an operating device that remotely operates the display device.
- the marker display control device of this aspect for example, when the marker can be detected, it is possible to reduce the difficulty in viewing the display screen by reducing the display size of the marker. Further, for example, when the marker cannot be detected, the marker can be easily detected by increasing the marker display size, changing the marker display position (moving the marker), or the like.
- the marker display control device of the present invention it is possible to reduce the difficulty of viewing the display screen of the display device while facilitating detection of the marker.
- the marker is used to specify the type of display device in the captured image, or to specify the position of the display screen of the display device.
- the display device to be operated by the user can be specified.
- the position of the display screen of the display device for example, the operation menu of the display device can be remotely operated by operating the photographed operation menu on the operating device at hand. Specifically, when the operation menu is displayed on the display device, the user designates the position of the operation menu of the display device in the captured image displayed on the display unit of the operation device, thereby displaying on the display device. You can send which coordinates on the screen are specified. Then, the display device can accept an operation on the assumption that the option displayed at the designated coordinates is selected.
- the display area of the marker can be changed, the display position of the marker can be changed, the form (for example, shape, color, etc.) of the marker can be changed.
- Changing the display area of the marker includes changing the size (display size) of the marker, changing the number of displayed markers, and the like.
- “Instruction to change the display mode of a marker based on the result of marker detection” means that, for example, when a marker is detected, the display area of the marker is reduced (the display size is reduced, the number of markers displayed is reduced). Decrease etc.) can be an instruction. Accordingly, the marker display area can be increased to facilitate marker detection before marker detection, and the marker display area can be appropriately reduced after marker detection.
- the “instruction to change the marker display mode based on the marker detection result” is, for example, an instruction to enlarge the marker or an instruction to change the marker display position when the marker is not detected. can do. By enlarging the marker, for example, even when the marker is not detected because the shooting distance is long, the marker can be detected. In addition, by changing the display position of the marker, for example, even when an obstacle is placed in front of the display device and a part of the display device (the part where the marker is displayed) cannot be captured, the marker is detected. The possibility of being increased.
- a predetermined signal may be transmitted to the display device, or the display size and display position of the marker may be transmitted to the display device.
- the captured image is encoded. That is, a marker can be detected from image information obtained by encoding an image.
- the display device may be a single display device having an image display function such as a television receiver or a notebook personal computer, or may include a display device and an image reproduction device.
- the image reproducing device can be configured by, for example, a DVD player (recorder), an HDD recorder, a video camera, a PC (personal computer), or the like.
- the display area of the display device is a two-dimensional area where the display screen of the display device is arranged.
- the display area and the display screen have the same size.
- the communication means can be configured by a communication device such as an infrared communication device or a wireless LAN (Local Area Network) device.
- the communication unit may be any device as long as it can transmit information in at least one direction, and may be referred to as a transmission unit (transmission device).
- transmission device transmission device
- the communication means may be referred to as an information transmission means (information transmission device).
- “By communication means” means “via communication means”.
- the marker display mode changing unit reduces the marker display area with respect to the display device displaying the marker detected by the marker detection unit. To give instructions.
- the display content of the display device is reduced by at least one of reducing the display size of the marker and reducing the number of displayed markers. can do. Therefore, for example, a relatively large marker can be displayed on the display screen of the display device, and the marker can be reduced after the marker is detected.
- the marker display mode changing unit gives an instruction to reduce the display area of the marker when a predetermined condition is satisfied. For example, this is a case where the size of the marker in the photographed image is larger than the lower limit size detected as the marker.
- the marker display control device further includes a captured image input unit that repeatedly captures a marker, and the marker detection unit extracts a marker from images sequentially obtained by the captured image input unit.
- the marker display mode changing unit performs a process of detecting the display device displaying the marker. An instruction to increase the display size of the marker or an instruction to change the display position of the marker can be given.
- the marker display size can be increased, and the marker display position can be By changing (moving the marker), detection of the marker can be facilitated.
- the photographed image input unit can be configured by, for example, a digital camera or a digital video camera.
- the marker display control device is further displayed by marker identification information including color information of the marker assigned to the display device, a display device associated with the marker, and the display device.
- the marker can be detected based on the marker identification information, and the display device to be operated can be recognized based on the information on the display device associated with the detected marker. Furthermore, based on the display size of the marker, a size smaller than the current display size can be designated when the marker is reduced, and a size larger than the current display size can be designated when the marker is enlarged.
- the marker information holding unit is provided in the marker display control device.
- the marker information holding unit may be provided in addition to the marker display control device.
- the marker display control apparatus can acquire marker information from other components having a marker information holding unit.
- markers are displayed at a plurality of spaced locations in the display area of the display device, and the marker display mode changing unit is displayed on the display device.
- a marker shielding state determination unit that determines that the marker is in a shielding state that cannot be detected by the marker detection unit due to the marker being shielded by the shielding object or because the marker is out of the imaging range, It is possible to give an instruction to change the display position of the marker determined to be in the shielding state by the marker shielding state determination unit.
- the marker in a shielded state or the like is moved, and the marker is easily detected. Therefore, it becomes easy to reduce the difficulty in viewing the display contents of the display device by arranging the markers at the peripheral edge of the display area and reducing the number of displayed markers.
- the shielded state includes, for example, a shielded state where the marker displayed on the display device is blocked by an obstacle and cannot be photographed, and a state outside the photographing range where the marker is located outside the photographing range and cannot be photographed.
- the marker display mode change unit sets the display position of the marker that is not in a shielded state to any of the four corners of the display area of the display device.
- An instruction may be given to change the display position of the marker in a shielded state or the like to the center side of at least one of the horizontal direction and the vertical direction of the display area.
- the marker is easily detected by bringing the marker displayed at the corner of the display area of the display device closer to the center.
- the marker display mode change unit detects the marker detected by the marker detection unit in the size of the marker in the image obtained by photographing the display content of the display device.
- the display device may be instructed to display the marker with a display size that is less than the size and greater than or equal to the lower limit size detected as the marker in the image.
- a lower limit size for detecting a marker in an image (captured image) obtained by capturing the display content of the display device is determined in advance. Therefore, the display size of the marker can be determined so as to be equal to or larger than the lower limit size in the captured image. As a result, for example, the marker display size can be sufficiently increased before marker detection, and the marker size can be reduced at a stroke after marker detection.
- the marker display mode change unit instructs the display device to reduce the display size of the marker until the detected marker is not detected, When no marker is detected, an instruction to increase the display size of the marker can be given.
- the marker can be reduced in stages even if the lower limit size is not fixed.
- the marker display mode change unit specifies the color of the marker to be displayed on the display device, and the marker detection unit images the display content of the display device In the image, a pixel group composed of pixels of the designated color and having a size equal to or larger than the lower limit size may be detected as a marker.
- the marker display control device further includes a coordinate conversion unit that converts coordinates in an image obtained by capturing the display content of the display device into coordinates in a display area of the display device, and the marker detection The unit may detect at least the position of the marker in the captured image, and the coordinate conversion unit may convert the coordinate based on at least the position of the marker.
- an operation menu is displayed on the display device, the operation menu is photographed with the operation device at hand, displayed on the display unit of the operation device, and the operation menu displayed on the display unit is operated. Is done. In such a situation, it is assumed that the position of the display device in the photographed image changes due to the hand shake or movement of the user, and it is preferable to detect the marker continuously and accurately detect the position of the display device.
- the coordinate conversion unit may perform coordinate conversion based on the positions of a plurality of markers, or may perform coordinate conversion based on the position of one marker and the size of the marker in the image.
- the display device has a function of displaying a display content including a marker and an operation menu in a display area
- the coordinate conversion unit includes: When coordinates on the display area of the display device in the image are designated, the coordinates can be converted into coordinates in the display area of the display device.
- the marker display control device designates a display unit that displays an image of the display content of the display device and arbitrary coordinates of the display unit.
- the display device has a function of displaying a display content including a marker and an operation menu in a display area, and the coordinate conversion unit is mounted on an operation device including an operation input unit in which operation input is performed.
- the coordinates of the display unit input to the operation input unit can be converted into coordinates in the display area of the display device.
- the marker display control device further includes a coordinate conversion unit that converts coordinates in an image obtained by capturing the display content of the display device to coordinates in a display area of the display device, and the shielding state
- a position calculating unit that calculates a position of the image before the change of the marker based on a change amount of the display position of the marker and the position of the image after the change of the marker
- the marker detection unit includes: The position of the marker in the image is detected, and the coordinate conversion unit converts the position of the marker that is not in the shielding state and the position before the change of the marker in the shielding state calculated by the position calculation unit. Conversion of the coordinates may be performed based on the above.
- An integrated circuit is connected to a display device by communication means, and performs control to display a recognition marker on the display device, and captures the display content of the display device.
- a marker detection unit that detects the recognition marker from the image, and a marker display mode change unit that instructs the display device displaying the marker to change the marker display mode based on the marker detection result It is characterized by comprising.
- a marker display control method is a method for performing control to display a recognition marker on a display device, the step of obtaining an image obtained by capturing the display content of the display device; And a step of instructing a display device displaying the marker to change the display mode of the marker based on the detection result of the marker.
- Embodiment 1 A marker display control device according to Embodiment 1 of the present invention, an operation device incorporating the marker display control device, and a display device will be described.
- a small information terminal equipped with a camera is taken as an example of an operation device, and a marker displayed on a display screen (also referred to as a display area) of a television that is an example of a display device by the small information terminal.
- a display screen also referred to as a display area
- a case will be described as an example where photographing and detection are performed and the size of the marker is adjusted.
- FIG. 1 shows a configuration of main parts of a small information terminal 1 incorporating the marker display control device 13 according to Embodiment 1 of the present invention and a television 2 (or personal computer 3) to be operated by the small information terminal 1.
- the small information terminal 1 is, for example, a small PC (personal computer) equipped with a camera, a mobile phone, a small music player, and the like, and includes a marker display control device and a configuration for controlling the recognized display device. It is a thing. Thereby, operation of the display apparatus image
- the display device is, for example, a television, a personal computer (hereinafter referred to as a PC), a liquid crystal projector, or the like, and has a configuration for displaying a marker. As a result, the type and position of the display device can be specified by the small information terminal 1.
- the display device is an example of a display device.
- the small information terminal 1 includes an operation input unit 10, a captured image input unit 11 (hereinafter sometimes referred to as “camera unit 11”), a display unit 12, a marker display control device 13, and a device operation control unit 14. And a transmission unit 15 (which is an example of a communication unit).
- an operation input unit 10 a captured image input unit 11 (hereinafter sometimes referred to as “camera unit 11”), a display unit 12, a marker display control device 13, and a device operation control unit 14.
- a transmission unit 15 which is an example of a communication unit).
- the operation input unit 10 includes, for example, input devices such as a keyboard, button switch, touch pad, and touch panel. When the operation input unit 10 receives a user operation input, the operation input unit 10 transmits the user operation content to the device operation control unit 14. In the example of FIG. 2, the operation input unit 10 is provided with a button switch 10 a and other switches.
- the photographed image input unit 11 is composed of a digital camera or the like. Also, the captured image input unit 11 captures the real world according to an instruction from the marker display control device 13, encodes an image captured of the real world, and generates captured image information. The captured image input unit 11 captures and generates captured image information several times to several tens of times per second, and outputs the captured image information to the display unit 12 and the marker detection unit 131.
- RGB888 color information is separated into luminance information of each color element of red, blue, and green, and the luminance value of each color element is expressed by 8 bits.
- the luminance value of the red element is expressed as Pr (x, y)
- the luminance value of the blue element is expressed as Pb (x, y)
- the luminance value of the green element is expressed as Pg (x, y). I will do it.
- the two-dimensional array P (x, y) expresses color information at coordinates (x, y), and the color information at each coordinate is called “pixel”.
- the display unit 12 includes a display panel such as a liquid crystal panel or an organic EL panel, and a display control unit that performs control to display an image on the display panel.
- the display unit 12 displays graphics, characters, An image including a landscape or the like is displayed.
- the display unit 12 displays a captured image obtained by imaging the captured image information generated by the captured image input unit 11 (so-called preview screen display is performed).
- the marker display control device 13 includes a marker detection unit 131, a display marker control unit 132, a marker information holding unit 133, and a marker display instruction unit 134. Further, the marker display control device 13 is configured to exhibit the function of the above configuration by executing a predetermined program by a microcomputer. The marker display control device 13 is formed in the integrated circuit. Here, the marker display control device 13 may not include the marker information holding unit 133.
- the marker display mode changing unit 130 includes a display marker control unit 132 and a marker display instruction unit 134.
- the marker detection unit 131 receives the captured image information from the captured image input unit 11 and detects a marker included in the captured image information.
- the marker detection unit 133 writes the marker information of the detected marker to the marker information holding unit 133 and notifies the display marker control unit 132 of the update of the marker information. Details of the marker information will be described later.
- the display marker control unit 132 extracts the marker information from the marker information holding unit 133, reduces the marker size within a marker detectable range based on the marker size detected by the marker detection unit 131, and displays the marker information holding unit. Write to 133. Further, the marker display instruction unit 134 is requested to instruct the television 2 displaying the marker to display the marker.
- the marker information holding unit 133 is a memory area (storage area) secured in a storage unit including a RAM, a nonvolatile memory (for example, a flash EEPROM), and holds marker information.
- a storage unit including a RAM, a nonvolatile memory (for example, a flash EEPROM), and holds marker information.
- the marker information includes information for identifying a marker, information on a display device associated with the marker, an instruction size of the marker, and the like.
- the information for identifying the marker is information relating to the morphological characteristics of the marker, and based on this information, the marker included in the captured image information can be detected.
- Examples of information for identifying a marker include “marker color”, “marker shape” and “marker pattern” in a two-dimensional marker.
- “marker color” is held in the marker information holding unit 133 as information for identifying a marker (see FIG. 7). That is, this is because markers of different colors are assigned to the television 2 and the PC 3 to identify them.
- the lower limit size of the marker included in the captured image information (lower limit value Pm of the number of pixels) is also held in the marker information holding unit 133 (not shown).
- the information on the display device associated with the marker is information on the display device that displays each marker. For example, the name of the television 2 (see FIG. 7), the IP address (not shown), and the like.
- the marker indication size is information for designating the size (display size) of the marker displayed on the display device.
- the marker information includes “marker detection information” which is information obtained by marker detection.
- the marker detection information includes information (FIG. 7) such as “detection flag”, “detection position”, and “detection size”, which will be described in detail later.
- the marker display instruction unit 134 reads marker information corresponding to the request from the marker information holding unit 133 based on a request from the display marker control unit 132, generates marker display instruction information, and sends the marker display instruction information to the transmission unit 15.
- the marker display instruction information is information configured by extracting information necessary for displaying the marker by the display device from the marker information.
- the information for identifying the marker and the instruction size of the marker are included in the marker display instruction information.
- the device operation control unit 14 specifies a display device to be operated based on information held by the marker information holding unit 133 when a marker is detected. Then, an operation menu 17 (for example, FIG. 2) corresponding to the specified display device is displayed on the display unit 12. Further, when a user input corresponding to the operation menu is made to the operation input unit 10, operation information indicating the contents of the user input is sent to the transmission unit 15 to the specified display device.
- an operation menu 17 for example, FIG. 2
- operation information indicating the contents of the user input is sent to the transmission unit 15 to the specified display device.
- the transmission unit 15 transmits information sent from the marker display instruction unit 134 and the device operation control unit 14 to the reception unit 21 of the television 2 by the communication means 5 such as infrared communication or wireless LAN (Local Area Network).
- the communication means 5 such as infrared communication or wireless LAN (Local Area Network).
- the connection relationship among the small information terminal 1, the television 2, and the PC 3 is schematically shown when the communication unit 5 is a wireless LAN.
- the communication means 5 is constituted by a master unit of a wireless LAN device.
- the communication means is constituted by the base unit of the wireless LAN device, the transmission unit 15, and the reception unit 16.
- the small information terminal 1 and the television 2 are directly connected by the communication means 5.
- the communication means includes the transmission unit 15 and the reception unit 21.
- the communication means includes the transmission unit 15 and the reception unit 21.
- PC3 is also provided with the same component.
- the receiving unit 21 receives the information transmitted by the transmitting unit 15 of the small information terminal 1 by the communication means 5 such as infrared communication, wireless LAN (or a wired LAN connected to the base unit of the wireless LAN device). Then, when the received information includes marker display instruction information, the receiving unit 21 sends the marker display instruction information to the marker display information control unit 22. On the other hand, when the received information includes operation information, the operation information is sent to the operation information receiving unit 24.
- the communication means 5 such as infrared communication, wireless LAN (or a wired LAN connected to the base unit of the wireless LAN device).
- the marker display information control unit 22 receives the marker display instruction information transmitted based on the request of the marker display instruction unit 134 of the small information terminal 1, and updates the marker display information holding unit 23 according to this information.
- the marker display information holding unit 23 is a memory area (storage area) secured in a storage unit including a RAM, a nonvolatile memory (flash EEPROM), and the like, and holds marker display information.
- the marker display information is information including the presence / absence of marker display, information on the display mode of the marker (for example, marker color, display size, etc.), and the contents are updated based on the marker display instruction information.
- the operation information receiving unit 24 receives the operation information transmitted based on the request from the device operation control unit 14 of the small information terminal 1 and inputs the operation information to the operation input unit 25.
- the operation input unit 25 includes any input device such as a keyboard, a button switch, a touch pad, and a touch panel. Further, the operation input unit 25 accepts operation information input from the operation information receiving unit 24 as user input.
- the content unit 26 generates display information to be displayed on the display unit 27 based on the information related to the user operation received from the operation input unit 25 and the content information (not shown).
- the content unit includes a tuner and a video processing circuit, and generates display information based on a video signal such as a television broadcast as content information.
- the content part is a part that performs functions of, for example, a WEB browser, a video player, a music player, and the like by executing predetermined application software.
- the WEB browser function acquires HTML data of a WEB page as content information through the Internet (not shown) based on a user operation, and the layout of display components such as images and buttons is based on the HTML data. Then, display information based on the layout and display components is generated and input to the display unit 27.
- the display unit 27 includes a display panel composed of a liquid crystal panel and the like, and a display control unit that performs control to display an image on the display panel, and displays the display information generated by the content unit 26.
- the display control unit has an OSD (on-screen display) function, and is superimposed on the display information generated by the content unit 26 based on the marker display information held by the marker display information holding unit 23 and is displayed on the display panel. Display the marker.
- the display control unit can generate display information such as an operation menu for setting the display device in accordance with a user operation.
- the information for displaying the marker is displayed with priority over the display information generated by the content unit 26. That is, an image based on the display information (corresponding to the display content inherent to the display device) is not displayed in the display area where the marker is displayed.
- the image based on the display information is a specific example of the “display device original display content”.
- FIG. 2 is a diagram showing an external configuration and arrangement of the small information terminal 1 that is an operation device and the television 2 and the PC 3 (personal computer 3) that are display devices in Embodiment 1 of the present invention.
- This figure shows a state in which the television 2 is photographed by the camera unit 11 (captured image input unit 11) mounted on the small information terminal 1.
- the small information terminal 1 displays a photographed image obtained by photographing the television 2 on the display unit 12 (specifically, the display area of the display unit 12 or the display screen of the display unit 12).
- the display unit 12 specifically, the display area of the display unit 12 or the display screen of the display unit 12.
- a marker 271a is displayed on the display unit 27 of the television 2 (specifically, the display area of the display unit 27 or the display screen of the display unit 27), and the captured image of the small information terminal 1 (that is, the display unit). 12), a marker photographed image 121a corresponding to the marker 271a appears.
- the PC 3 is installed outside the photographing range of the small information terminal 1, a portion corresponding to the PC 3 does not appear in the photographed image.
- FIG. 2 shows a state in which the operation menu 17 (including the operation content display image 17a) is displayed by the device operation control unit 14 after the marker 271a is detected by the marker detection unit 131.
- the operation input unit 10 of the small information terminal 1 includes a plurality of button switches.
- the button switch 10a accepts an input from the user who selects the operation content display image 17a displayed on the display unit 12. That is, when the button switch 10 a is pressed, operation information associated with the operation content display image 17 a is sent to the transmission unit 15.
- other button switches are associated with operation information or other operation contents.
- FIG. 3 is a flowchart showing display device control processing executed by the small information terminal 1.
- FIG. 4 is a flowchart showing instruction reception processing of the television 2 (or PC 3).
- the user activates application software for performing display device control processing by operating the small information terminal 1, and then displays the camera unit 11 (captured image input unit 11) in the direction of the television 2 and displays it on the display unit 12. It is assumed that the small information terminal 1 is held so that the television 2 appears in the captured image (FIG. 2).
- step S1 the marker information holding unit 133 is initialized, and a marker display command is issued to all display devices (in this example, the television 2 and the PC 3).
- the initialization of the marker information holding unit 133 is performed by the display marker control unit 132.
- FIG. 7 shows marker information held in the initialized marker information holding unit 133, and marker information and marker initialization processing will be described based on this figure.
- “device name”, which is information about the display device, is set.
- the color “red” is set for the device name “TV”
- the color “blue” is set for the device name “PC” so that the TV 2 and the PC 3 are the detection targets.
- the settings at the time of initialization are set in advance by setting at the time of product shipment, user registration, or the like, and those stored in the storage unit of the small information terminal 1 are used.
- detection flag is a flag indicating whether or not the marker detection unit has detected a marker, and “undetected” is set as an initial value.
- the detection position is the coordinate (Xd, Yd) of the marker detected by the marker detection unit 131 in the captured image information. Basically, the marker is not composed of a single pixel, but is detected as a pixel group in which a plurality of pixels are combined. For this reason, a problem arises as to which of the coordinates of the plurality of pixels is adopted as the detection position. In the present embodiment, the coordinate of the pixel located at the upper left among the plurality of pixels is employed.
- the detection position is set to a valid value only when the value of the detection flag is “detection”. In this initialization process, since the value of the detection flag is “not detected”, this value is “not set”.
- the detection size indicates the size of a plurality of pixels constituting the marker, and is expressed as (Wd, Hd) as a horizontal width Wd and a vertical width Hd.
- the horizontal width Wd and the vertical width Hd are the number of pixels in the X-axis direction and the number of pixels in the Y-axis direction, respectively, of a quadrangle (for example, a rectangle) formed by the pixel group constituting the marker.
- FIG. 8A is a diagram illustrating the photographed image 111
- FIG. 8B is a diagram illustrating the region 111a in an enlarged manner.
- a region 111a in FIG. 8A is a region including a portion (pixel group 111b) corresponding to the marker displayed on the television 2 in the captured image 111, and the region 111a is shown in an enlarged manner in FIG. 8B.
- Each small square in the region 111a represents a pixel.
- a black pixel group 111b indicates a portion (marker photographed image) corresponding to the marker displayed on the television 2 in the region 111a.
- the detection size is (33, 30).
- this detection size is set to a valid value only when the value of the detection flag is “detection”. In this initialization process, since the value of the detection flag is “not detected”, this value is “not set”.
- the designated size designates the size (display size) of the marker displayed by each display device, and is expressed as (Wr, Hr) as the horizontal width Wr and the vertical width Hr.
- the horizontal width Wr and the vertical width Hr are the number of pixels in the X-axis direction and the number of pixels in the Y-axis direction of a rectangle formed by the pixel group constituting the marker, as in the detection size.
- the initial value is set to be large, and the marker can be detected even when the distance between the small information terminal 1 and the television 2 is relatively long.
- the marker display command to all display devices in step S1 is made by transmitting the marker information set in the initialization process to the television 2 and the PC 3 as marker display instruction information via the communication means 5.
- the marker display instruction information transmission process is schematically shown by a broken line.
- the transmission of the marker display instruction information is assumed to follow the symbol “A” in FIG. 4 from the symbol “A” in FIG.
- the transmission of other marker display instruction information (S4) and the transmission of operation information (S5) are similarly shown.
- step S1 for example, when three or more display devices are set in the marker information holding unit 133, marker display instruction information is transmitted to all three or more display devices. That is, marker display instruction information is transmitted to all display devices set in the marker information holding unit 133. Thereby, it is possible to instruct the display device to display the marker even in a state before the marker detection is performed.
- the marker display instruction information in this example is information composed of “color” and “instruction size”. That is, in the marker display instruction information of this example, information that the color is “red” and the instruction size is (150, 150) is transmitted to the television 2, and the color is “blue” and the instruction size is (150, 150). 150) is included.
- step S2 the captured image input unit 11 (camera unit 11) captures the television 2 to generate captured image information, and inputs the generated captured image information to the marker detection unit 131.
- the captured image input unit 11 also inputs the actual captured image information to the display unit 12.
- the display unit 12 converts the input captured image information into an image and displays the captured image.
- This step S2 and steps S3 to S6 described later are performed at a frequency of several to several tens of times per second. Therefore, a real-world landscape photographed by the photographed image input unit 11 at a frame rate of several frames to several tens of frames per second is displayed on the display unit 12 as a moving image.
- the marker detection unit 131 performs a process of detecting a marker included in the captured image information, a process of updating the marker information, and the like.
- the marker detection unit 131 detects a marker
- the marker detection unit 131 acquires marker detection information such as a marker detection size.
- the marker information held in the marker information holding unit 133 is updated by writing the marker detection information in the marker information holding unit 133.
- FIG. 9 is a diagram illustrating an example of the contents (held marker information) of the marker information holding unit 133 after the marker detection process.
- the marker displayed on the television 2 is detected, the detection flag is set to “detection”, and the detection position and the detection size are set.
- the display marker control unit 132 reduces the marker instruction size based on the marker detection size and the like, and updates the marker information. Then, the marker display instruction unit 134 generates marker display instruction information including the reduced instruction size and transmits it to the television 2.
- FIG. 10 is a diagram illustrating an example of the contents of the marker information holding unit 133 after the marker size adjustment processing in step S4.
- the instruction size is reduced to (23, 25) by the marker size adjustment processing.
- the detection size (5, 5) is set as the detection size when a marker whose instruction size is reduced to (23, 25) is photographed and detection processing is performed.
- step S5 the display device to be operated by the user is determined, the operation menu 17 is displayed, the user input is received, and the user operation content is transmitted.
- the device operation control unit 14 determines a display device to be operated by the user based on the marker information detected in step S3. Then, the device operation control unit 14 displays the operation menu 17 associated with the determined operation target display device on the display unit 12 and waits for an input of a user operation. Thereafter, when a user operation corresponding to the operation menu 17 is input to the operation input unit 10, operation information indicating the user's operation content is transmitted to the operation target display device via the transmission unit 15.
- the device operation control unit 14 determines a display device to transmit the operation content selected by the user input as operation information based on the marker information held in the marker information holding unit 133.
- the display device related to the marker information whose detection flag value is “detection” is determined as the display device to which the operation information is to be transmitted. Therefore, when the marker information holding unit 133 has the contents shown in FIG. 9, the operation target display device is the device name “TV” whose detection flag is “detection”, that is, the television 2.
- a menu related to device selection may be displayed on the display unit 12, and the operation target display device may be determined by user input.
- the marker information holding unit 133 can store whether or not the user has selected each display device. Then, the display device selected by the user can be determined as the display device to be operated.
- the small information terminal 1 includes a storage unit (not shown), and operation menu image information associated with each display device is stored in the storage unit. Then, after determining the operation target display device, the device operation control unit 14 extracts the operation menu image information corresponding to the operation target display device from the storage unit, and transmits it to the display unit 12 together with an operation menu display command. As a result, the operation menu 17 is displayed.
- the device operation control unit 14 receives a user input from the operation input unit 10.
- the user presses the button switch 10a.
- an operation for changing the channel to 2ch in the broadcast program viewing content is input to the button switch 10a as shown in the operation content display image 17a.
- Information is associated. Therefore, the operation content selected by the received user input is a “2ch” operation that is the content displayed on the operation content display image 17a, that is, an operation for changing the channel in the program viewing content to 2ch.
- the device operation control unit 14 requests the transmission unit 15 to “transmit the operation content input by the user to the operation target display device”.
- operation information “change channel to 2ch” is transmitted to the television 2.
- this process S5 is synchronized with the marker detection process S3 and the marker size adjustment process S4, but may be an asynchronous process.
- operation menu 17 shows an example of displaying channel operations
- various other menus can be displayed. For example, options such as volume adjustment and input switching may be displayed. In this case, a plurality of operations such as channel change and volume adjustment can be sequentially performed.
- step S6 The processes from S2 to S5 are repeatedly performed until the display device control process is completed.
- the display device control process ends when, for example, an application software end operation for performing the control process is performed by a user input.
- a marker display termination instruction is transmitted to all display devices (such as the television 2), and the marker display on all the display devices is terminated.
- the display device control process may be terminated or suspended when the operation is not input even after the set time has elapsed.
- the temporary stop is a state in which display device control processing is not performed until a restart operation is performed.
- the marker 2 is displayed on the television 2.
- FIG. 4 is a flowchart showing the processing of the television 2 (or PC 3) performed on the marker display instruction information transmitted from the small information terminal 1 or the like. 4 representatively shows the processing of the television 2, the processing of the PC 3 is the same.
- the marker display information control unit 22 initializes the contents (not shown) of the marker display information holding unit 23.
- the marker display information includes at least a “display flag”, “color”, and “instruction size”.
- the “display flag” is information indicating whether to display a marker.
- the color and the instruction size are the same as those described in step S1.
- the display flag is set to “non-display”, “color” and “instruction size” are set to “not set”.
- step S12 the reception unit 21 waits for reception of marker display instruction information and operation information (input from the symbol “A” in FIG. 3) transmitted from the transmission unit 15 of the small information terminal 1.
- step S13 When the receiving unit 21 receives the operation information (step S13), the operating state of the display device is changed according to the operation information (step S14). This will be specifically described below.
- the operation information received by the receiving unit 21 is input to the operation information receiving unit 24.
- the operation information receiving unit 24 changes the operation information into a user operation event format in the television 2 and inputs the operation information to the operation input unit 25.
- the operation input unit 25 inputs the input user operation event to the content unit 26.
- the content unit 26 performs processing according to this event and generates display information.
- the television 2 that has received the “2ch” operation signal shown in step S4 changes the channel to “2ch” and generates display information of the program viewing content broadcast on the channel. .
- the display information is converted into an image and displayed on the display unit 12, and the operation of changing the channel is completed.
- step S15 When the receiving unit 21 receives the marker display instruction information in step S12 (step S15), the marker display information is updated (step S16). This will be specifically described below.
- the marker display instruction information received by the receiving unit 21 is input to the marker display information control unit 22.
- the marker display information control unit 22 updates the contents of the marker display information holding unit 23 according to the marker display instruction information.
- the television 2 updates the color to “red” and the instruction size to (150, 150), and the PC 3 displays the color “blue”.
- the size is updated to (150, 150).
- the display flag value is switched to “display” in accordance with reception of this information.
- step S1 is executed only once and the marker display instruction information is transmitted only once to the display device, whereas the process of step S4 (details will be described later) is repeatedly executed and necessary. Accordingly, the marker display instruction information is sent out a plurality of times, so that the updating process may be carried out a plurality of times.
- the television 2 is set so as to display the square marker having the indicated size, but the shape of the marker may be designated by the small information terminal 1.
- step S17 the display unit 27 displays the marker superimposed on the display information generated by the content unit 26.
- the marker to be displayed in a superimposed manner follows the contents of the marker display information holding unit 23. That is, when the display flag of the marker display information is “display”, the display unit 27 draws the marker on the display information in accordance with the color and the value of the instruction size.
- the television 2 draws a red square having a width of 150 pixels and a height of 150 pixels on the display information
- the PC 3 draws a blue square having a width of 150 pixels and a height of 150 pixels on the display information.
- the display device determines an arbitrary fixed position (for example, the coordinate (0, 0) is the upper left coordinate of the marker) and displays it at this position.
- the display instruction position may be included in the marker display information, and the marker may be displayed at the display instruction position (this configuration will be described in detail in Embodiment 3).
- step S18 The above process is repeatedly performed at least while the display device control process of the small information terminal 1 is continued (step S18).
- step S11 the end instruction
- a marker display is hidden (OFF) by the initialization process of step S11. Thereafter, reception of a display instruction or the like is performed (S12).
- step S3 the marker detection unit 131 receives the captured image information generated by the captured image input unit 11, and detects a marker included in the actual captured image information.
- the captured image information generated by the captured image input unit 11 in step S2 is information of the captured image 111 illustrated in FIG.
- step S31 a part of the marker information held by the marker information holding unit 133 is initialized. Specifically, the marker information detection flag related to all display devices held by the marker information holding unit 133 is reset to the “not detected” state. Also, the detection position and detection size values are set to “not set”.
- step S32 the marker color assigned to each display device is read from the marker information holding unit 133 and acquired.
- “red” of the television 2 and “blue” of the PC 3 are acquired as shown in FIG.
- step S33 a pixel group of “red” and “blue” which are the colors read in step S32 is detected from the photographed image information.
- the detection of the pixel group of each color is performed as follows based on the color information of each pixel P (x, y).
- the luminance difference total value C (i) of each color component of RGB (red, green, blue) with respect to each reference color is obtained by the following equation.
- Pr (x, y), Pg (x, y), and Pb (x, y) are the luminance values of the red, green, and blue components of each pixel, respectively.
- the reference luminance values Sr (i), Sg (i), and Sb (i) are the luminance values of red, green, and blue components that constitute the reference color, respectively.
- the reference luminance value for each color is stored in the storage unit of the small information terminal 1.
- the luminance difference total value C (i) of each pixel of the captured image information is calculated.
- N 8 in the example of FIG. 11, eight luminance difference total values C (1) to C (8) are calculated for each pixel.
- the color name Cn (i 0 ) corresponding to the color number i 0 of the smallest of the eight luminance difference total values C (1) to C (8) is the color name of the pixel.
- the pixel groups are P (x k ⁇ 1, y k ), P (x k +1, y k ), P (x k , y k ⁇ 1), and P (x k , y k +1).
- color name corresponding to any one or more pixels of) is a set of P (x k, y k) of the corresponding color names the same as comprising the pixel P (x k, y k) .
- the subscript k is, for example, a positive integer
- P (x k , y k ) represents the color information of the k th coordinate.
- each pixel constituting the portion 111b where the marker displayed on the television 2 is captured should have a luminance value close to the color name “red”. Therefore, in this step, the area 111b is found as a pixel group having the color name “red”.
- an upper threshold value may be provided for C (i) so that a color name as close to the reference color as possible is selected.
- the color name of the pixel is not set (for example, the color number is set to 0). Is done.
- a color having a reference luminance value that is significantly different from the reference color is excluded.
- only a pixel group constituted by pixels whose minimum value among the luminance difference total values C (1) to C (N) is equal to or less than the upper threshold is set as a marker detection target. By doing in this way, the precision of marker detection can be improved.
- step S34 it is determined whether the pixel group detected in step S33 is a marker. Specifically, the determination is made based on whether or not the number of pixels in the pixel group is equal to or greater than the lower limit value Pm. That is, it is determined that a pixel having a number of pixels less than the lower limit Pm is not a marker but noise.
- the lower limit Pm is calculated by (Equation 2), and the minimum value of the lower limit Pm is 1.
- the set value Wp is the number of pixels in the horizontal direction
- the set value Hp is the number of pixels in the vertical direction.
- Both the set values Wp and Hp are set to values that can reduce the marker within a range where the marker can be detected. In the present embodiment, for example, the set values Wp and Hp are both “5”.
- the coefficient ⁇ is a value that is set to be greater than “0” and equal to or less than “1”. The smaller this value is, the more reliably the marker can be detected. However, a small pixel group is also detected as a marker, and the possibility of erroneous detection using noise as a marker increases. On the other hand, if the coefficient ⁇ is increased, the possibility of erroneous detection of noise as a marker decreases, but there is a possibility that a small marker may be excluded as noise. Therefore, in order to reduce false detection, the coefficient ⁇ is preferably set to 0.5 or more. However, in a situation where erroneous detection is unlikely to occur, the coefficient ⁇ can be less than 0.5.
- the coefficient ⁇ can be determined based on, for example, the performance (for example, resolution) of the camera unit 11 (captured image input unit 11). If the pixel performance of the camera unit 11 is low, for example, even if the marker is displayed on the television 2 with the display size calculated in step S33 described later, the size of the photographed marker may be smaller than the calculated value. . In order to prepare for such a situation, it is preferable to set the coefficient ⁇ to a certain small value, for example, 0.6.
- the coefficient ⁇ is determined, for example, by performing test photographing at the time of product shipment or when executing application software that performs the above-described display device control processing.
- the number of pixels ( ⁇ ⁇ Wp, ⁇ ⁇ Hp) obtained by multiplying the number of pixels Wp and Hp by the coefficient ⁇ is the lower limit size (minimum size necessary for marker detection) detected as a marker.
- the reference for determining the pixel group as a marker is changed to the set value ( ⁇ It may be greater than or equal to Wp, ⁇ ⁇ Hp).
- step S35 for the pixel group determined to be a marker in step S34, the marker information related to the pixel group is written into the marker information holding unit 133.
- the information written here is the detection position and detection size of the pixel group.
- the detection flag is “detection”.
- FIG. 6 is a flowchart showing the marker size adjustment processing in step S4. Based on this figure, the marker size adjustment processing will be described.
- step S41 the display marker control unit 132 extracts the marker information of each display device detected by the marker detection unit 131 from the marker information holding unit 133, that is, the marker information in which the value of the detection flag is detected. In this example, only marker information relating to the television 2 is extracted.
- step S42 the display marker control unit 132 calculates a display size that can reduce the marker within a range that can be detected by the marker detection unit 131, based on the marker information acquired in step S41.
- the display size (Wm, Hm) of the marker at the time of reduction is calculated by the equation shown in (Expression 3).
- the set value Wp of the number of pixels in the horizontal direction capable of detecting the marker, the set value Hp of the number of pixels in the vertical direction, the detection size of the marker information is (Wd, Hd), and the indicated size of the marker information is (Ws, Hs) ).
- the indicated size of the marker information is updated to the calculated size (23, 25) (FIG. 10).
- step S43 the reduced instruction size is written in the marker information holding unit 133, and a marker display instruction is given to the television 2.
- the marker display instruction unit 134 transmits marker display instruction information to the television 2 in which the instruction size has been updated in step S42.
- the marker display instruction information has a color “red” and an instruction size (23, 25).
- the marker 271a that is displayed large on the television 2 as shown in FIG. 2 can be reduced and displayed on the television 2 as shown in FIG. Therefore, it is possible to reduce the difficulty in viewing the original display content of the television 2.
- the display size of the marker 271a is calculated by the above equation 3, so that the instruction size is calculated in the next marker detection process.
- the marker size is (Wp, Hp).
- This detection size is detected as a marker because the number of pixels in the pixel group constituting the marker is equal to or greater than the aforementioned lower limit value Pm.
- the lower limit value Pm is a coefficient ⁇ ⁇ Wp ⁇ Hp (0 ⁇ ⁇ 1). In this way, by determining the lower limit value Pm and determining the instruction size so that the marker size in the captured image becomes (Wp, Hp), the instruction size becomes a value with which the marker can be detected almost certainly. . Thus, the marker instruction size can be reduced as much as possible within the range where marker detection is possible.
- the coefficient ⁇ is determined based on the performance of the camera unit 11, while the set values Wp and Hp are not affected by the performance of the camera unit 11. That is, by calculating the instruction size for reducing the marker based on the setting values Wp and Hp, the instruction size can be calculated without considering the pixel performance of the camera unit 11.
- the small-sized information terminal 1 can recognize the marker 271a that is largely displayed on the television 2 as shown in FIG. 2 as shown in FIG. Therefore, the marker 271a can be changed to the smallest possible size.
- the size of a marker displayed on a display device such as a television is made small enough to enable detection, and marker detection is facilitated. Furthermore, by making the marker display size relatively large in a state before performing the marker detection process, the marker is not easily affected by the shooting distance, and the marker can be easily detected. That is, it is possible to reduce the difficulty in viewing the display content inherent to the display device while ensuring the ease of marker detection.
- the shape of the marker is not limited to a quadrangle, and may be, for example, a triangle, a circle, an L shape (a bowl shape), a belt shape, or the like.
- the marker shape is a right triangle or L shape, a right angle corner can be arranged at the corner of the display screen of the display device to effectively use the display screen.
- the display area may be greatly increased.
- it can be displayed on the entire display screen (also referred to as a display area) of the display device, or can have a frame shape along the outer periphery of the display screen. In that case, initial marker detection becomes easy.
- a marker may be configured by combining a plurality of colors.
- one marker can be divided into two or divided into three or more, and two or more colors can be assigned to one marker.
- one marker can be configured to surround the first color portion with the second color portion. Even in this case, the accuracy of marker detection is improved.
- the contour of the first color portion is accurately reproduced in the captured image, and pattern matching is performed. It is suitable for the case.
- the marker may display a symbol in a frame.
- the figure in this frame is an example of a marker pattern.
- the number of markers is not limited to one, and a plurality of markers can be displayed.
- markers may be displayed at two corners located on the diagonal of the display screen of the display device.
- a preset color is designated as the marker color.
- a procedure of selecting a color that is not included in the current photographed image information, that is, an unused color can be adopted.
- FIG. 13 shows a flowchart of an unused color search process for searching for an unused color in the captured image information.
- captured image information obtained by photographing the television 2 or the like without displaying the marker on the television 2 or the like, or the number of markers detected in the marker detection process is larger than the number of markers displayed. This is performed on photographed image information and the like.
- steps S1002 to S1007 the presence / absence of a pixel group having the number of pixels equal to or greater than the threshold value Pn is checked for the reference color having a color number of 1 to N (N is, for example, 8).
- the threshold value Pn is set to a value equal to or lower than the lower limit value Pm.
- unused colors are assigned to the markers in S1008.
- the marker information is updated, and the unused color is set as the marker color for each display device.
- step S1 or S3 the marker of the color assigned to the television 2 or the like is displayed.
- FIG. 4 Steps S15 to S17.
- markers of colors that are not used in the captured image information are displayed on the television 2, thereby facilitating marker detection. Can do.
- steps S1002 and S1003 it may be determined whether or not there is a pixel having the same color as each reference color. If a reference color having no pixel of the same color in the captured image information is assigned to the television 2 or the like, the marker detection accuracy can be further improved. As described above, if the marker of each display device is displayed in a color not included in the captured image information, the color does not exist other than the marker, and therefore step S34 may be omitted.
- steps S1002 to S1007 it is not essential to check whether or not all the reference colors are not used. For example, it is only necessary to detect the same number of unused colors as the number of registered display devices. Further, in the case where erroneous marker detection has occurred, it is not necessary to check the color of the erroneously detected marker.
- FIG. 14 shows a flowchart of the marker detection process 2.
- the processes from step S31 to S35 are the same as the marker detection process 1, and thus the description thereof is omitted.
- step S36 it is determined whether or not the number of pixel groups (number of pixels is Pm or more) regarded as markers, that is, the number of detected markers is greater than the number of displayed markers (one in the first embodiment). The It is assumed that the number of markers displayed is indicated by the small information terminal 1 and stored in the small information terminal 1.
- step S37 the above-described unused color search process is performed in step S37, and the marker color assignment to each display device is changed.
- the unused color search process can also be performed before the marker initialization process (S1 in FIG. 3).
- the television 2 or the like can be photographed in a state where the marker is not displayed by a user operation, and the unused color search process can be performed on the photographed image information obtained by the photographing.
- the color assigned to each display device in the unused color search process is set in the marker information holding unit 133 in the marker initialization process (S1), and is transmitted to the television 2 or the like by the marker display instruction information.
- a process or the like for detecting a pixel or a group of pixels whose luminance difference total value C (i) is not more than a threshold value may be performed.
- the marker detection process it is preferable to detect a pixel group in which the luminance difference total value C (i) is equal to or less than a threshold value.
- the color of the marker can be simply changed without performing the unused color search process. In this case, by changing the color of the marker one after another until the erroneous detection of the marker does not occur, an unused color is eventually assigned to the marker.
- pattern matching is performed on the pixel group detected in S33, and it can be determined whether or not it is a marker.
- pattern matching the feature amount between the marker shape data and the shape of the pixel group is compared, and when the difference between the feature amounts is equal to or less than a set value, it is determined that the marker is a marker. Therefore, even if the pixel group having a shape that is not approximate to the marker shape is the same color as the marker, it is determined that the pixel group is not a marker.
- the accuracy of marker detection can be improved by performing the above pattern matching.
- the luminance difference total value C (i) is calculated by the marker detection process. Instead of calculating the luminance difference total value C (i), whether or not each pixel is the same color as the reference color can be determined depending on whether or not the luminance condition of each reference color is satisfied.
- the luminance threshold values Br (2), Bg (2), and Bb (2) are set for each of RGB.
- the red luminance value Pr (x, y) is equal to or higher than Br (2)
- the green luminance value Pg (x, y) is equal to or lower than Bg (2)
- the blue luminance value Pb (x, y) is equal to Bb (2).
- Pixels having the same color as the marker can be extracted by detecting the following pixels.
- Each luminance threshold value can be, for example, 200 for Br (2) and 30 for Bg (2) and Bb (2).
- the luminance threshold is used as the lower limit for the color component with the reference luminance value of 255
- the luminance threshold is used as the upper limit for the color component with the reference luminance value of 0. It is done.
- FIG. 15 shows a small information terminal 500 that is an operating device in which the marker display control device according to Embodiment 2 of the present invention is incorporated, and a television 2 (PC 3) that is a display device to be operated by the operating device.
- FIG. 15 the same components as those in the first embodiment are denoted by the same reference numerals as those in FIG. 1, and the description thereof is omitted or simplified.
- symbol as FIG. 1 may have the function which the component of FIG. 1 does not have.
- the marker display control device 513 in FIG. 15 includes a non-detectable marker control unit 135 in addition to the same configuration as the marker display control device 13 in FIG. 1 described in the first embodiment.
- the undetectable marker is a marker that was detected in the previous (or several times) previous marker detection process and could not be detected in the current marker detection process.
- the non-detectable marker control unit 135 performs a process of increasing the display size of the marker for the non-detectable marker in order to detect the marker again.
- FIG. 16 shows the setting contents of the marker information holding unit 133 in the state shown in FIG.
- an item “previous detection position” is added in order to determine a marker that cannot be detected.
- FIG. 17 is a diagram showing the external configuration and arrangement of the operation device (small information terminal 500) and the display device (TV 2, PC 3) in Embodiment 2 of the present invention.
- This figure shows a state in which the small information terminal 500 is moved away from the television 2 from the arrangement of FIG.
- the marker photographed image 121b which is a portion corresponding to the marker 271b on the small information terminal 500, falls below the lower limit value Pm of the number of pixels necessary to determine the above-described pixel group as a marker. That is, in the process of step S34, the marker photographed image 121a is detected as a marker in the state of FIG. 2, but the marker photographed image 121b is not determined as a marker in the state of FIG.
- FIG. 18 is a flowchart of the marker size adjustment process 2.
- the marker size adjustment process 2 is obtained by adding the undetectable marker display mode changing process (hereinafter referred to as undetectable marker process) in step S44 to the marker size adjustment process (S4) of the first embodiment. is there. Steps S41 and S42 are the same as those in the first embodiment. If no marker is detected, no processing is substantially performed in steps S41 and S42.
- the non-detectable marker control unit 135 increases the marker display size so that the non-detectable marker can be detected according to the procedure shown in FIG. Perform the process.
- an undetectable marker is specified.
- Requirement 1 is that the detection position of the marker information held by the marker information holding unit 133 is “not set”. This is to indicate that the marker related to the marker information has not been detected in the current marker detection process. In requirement 1, the detection flag may not be detected.
- Requirement 2 is that the previous detection position is not “not set” in the marker information. This is to indicate that the marker related to the marker information has been detected at the detection position in the previous marker detection process. In this example, as described in FIG. 16 above, the marker related to the television 2 is specified as a non-detectable marker.
- step S443 If the undetectable marker cannot be specified in step S441, the process of S443 is not performed (step S442).
- step S444 the value of the detection position of each marker information is set to the previous detection position.
- the set value of the previous detection position is set to the detection position detected in the previous (that is, current) marker detection process in the next undetectable marker process.
- the value of “Detection position” when the value of “Detection position” is not set, the value of “Previous detection position” is not changed. That is, when the value of “detection position” is not set and some coordinates are set at the previous detection position, the coordinates are held.
- the instruction size is transmitted to the television 2 as marker display instruction information (S43).
- the display size of the marker displayed on the television 2 is increased (S15 to S17), and the subsequent marker detection (S3) is facilitated.
- this step S44 causes the marker size to be sequentially updated to a larger value until the undetectable marker can be detected. After the marker can be detected, the marker can be detected and adjusted to the minimum size according to the procedure described in the first embodiment.
- the marker 271b is detected.
- the display size of the marker displayed on the display device such as the television 2 by the display control device 513 can be increased to detect the marker.
- the orientation of the camera unit 11 of the small information terminal 500 may be changed by the user, and the display device may be located outside the shooting range. In preparation for such a situation, the following processing can also be performed.
- step S443 for the undetectable marker, the number of instructions for increasing the instruction size is counted, and if the marker is not detected even if the increase instruction is performed a set number of times after S444, for example, the television 2 in the shooting range It is also possible to provide a step of determining that the non-detectable marker process is terminated. Further, a step of ending the non-detectable marker process may be provided when the instruction size for the non-detectable marker exceeds the upper limit set value. Note that a step of performing the above processing may be provided after S444.
- the “previous detection position” of the non-detectable marker is not set, and the non-detectable marker is not specified.
- an instruction to end the display of the marker or an instruction to reduce the instruction size is given to the display device displaying the undetectable marker.
- an instruction to reduce the instruction size is given, for example, the instruction size before the undetectable marker is specified is stored in the marker information holding unit 133, and the instruction size can be used.
- the marker display mode changing unit 530 includes a display marker control unit 132, a marker information holding unit 133, a marker display instruction unit 134, and a non-detectable marker control unit 135.
- the marker detection process is performed while gradually reducing the display size, and when the marker is no longer detected, the display size of the marker can be detected. It is also possible to perform control to adjust to.
- an item “previous instruction size” is added to the marker information stored in the marker information holding unit 133.
- the designated size can be returned to the value of the previous designated size.
- the indicated size of the marker is reduced by about 10%, for example.
- the instruction size is reduced from (150, 150) to (135, 135).
- the original instruction size for example, (150, 150) is copied to the previous instruction size.
- the designated size is set to the value of the previous designated size in S443.
- the display size of the marker can be made as small as possible and detectable. Note that after the instruction size is set to the value of the previous instruction size, the process of reducing the marker instruction size is not performed.
- the increase in the instruction size can be reduced (for example, 1.1 times). In this case, it is possible to prevent the marker from becoming excessively large, and it is possible to prevent the display content inherent to the display device from becoming difficult to see.
- a plurality of markers may be displayed on each display device. If there are a plurality of undetectable markers, the designated size of each undetectable marker is increased in S44.
- the television 2 displayed on the display unit 12 is specified by designating coordinates on the display unit 12 of the small information terminal 600 (specifically, the display screen or display area of the display unit 12). It is possible to remotely control the operation menu.
- FIG. 20 is a diagram illustrating a configuration of a small information terminal 600 that is an operation device incorporating the marker display control device according to Embodiment 3 of the present invention, and a television 2 and a PC 3 that are display devices to be operated by the small information terminal 600.
- the same components as those in FIGS. 1 and 15 are denoted by the same reference numerals, and description thereof will be omitted or simplified.
- symbol as FIG. 1 etc. may have the function which does not exist in the component of FIG.
- the marker display control device 613 in FIG. 20 includes a marker shielding determination unit 136 and a display device information generation unit 137 in addition to the configuration of the marker display control device 513 in FIG. 15 described in the second embodiment.
- the marker display mode changing unit 630 includes a marker shielding determination unit 136.
- FIG. 21 is a diagram showing the television 2 in a state where the four markers 271d to 274d and the operation menu 28 are displayed, and the small information terminal 600 that photographs the television 2.
- the television 2 (and the PC 3) has a function of displaying an operation menu 28 for performing image quality setting and the like.
- the operation menu 28 includes three operation content display images 28a, b, and c. Normally, one of the operation content display images 28a, 28b, and 28c is selected and the image quality setting operation is performed by a remote control operation dedicated to the television.
- the operation menu 28 is formed using a color (for example, an intermediate color) different from the color used for the marker. By doing in this way, detection of a marker becomes easy. Note that the color used for the operation menu 28 in each display device can be held in the marker information holding unit 133 as marker information, and the other colors can be used as the marker colors.
- the operation input unit 10 of the small information terminal 600 includes a touch panel that covers the liquid crystal panel of the display unit 12. Therefore, the user can input arbitrary coordinates Qa (Xa, Ya) on the display unit 12 by touching the touch panel. That is, the coordinates Qa (Xa, Ya) on the display unit 12 can be designated.
- step S5 the display device information generation unit 137 has a coordinate position Qa (Xa, Ya) on the display device in the captured image information (hereinafter sometimes referred to as a captured image coordinate position Qa). Is converted into a coordinate position Qb (Xb, Yb) in the display area of the actual display device (hereinafter, sometimes referred to as display area coordinate position Qb).
- generating the display area coordinate position Qb by coordinate conversion of the captured image coordinate position Qa is expressed as “generation of point information Qb (Xb, Yb)”.
- the point information Qb (Xb, Yb) is coordinate information indicating which part of the display area of the display device the coordinate Qa (Xa, Ya) in the captured image information corresponds to.
- the generation of the point information Qb (Xb, Yb) will be described in detail with reference to FIG.
- the display unit 12 in FIG. 21 displays the captured image as it is. Further, the display unit 12 has a so-called touch panel mounted on the liquid crystal panel as described above, and also serves as the operation input unit 10. That is, it is assumed that the user can designate one point Qa (Xa, Ya) on the captured image by a touch operation.
- the point information Qb (Xb, Yb) of the television 2 is calculated by the equation exemplified in (Equation 4).
- the number of pixels in the horizontal direction of the display area of the television 2 in the captured image is indicated by Wa
- the number of pixels in the vertical direction is indicated by Ha
- the number of pixels in the horizontal direction of the display area of the actual television 2 is indicated by Wb
- the number of pixels in the vertical direction is indicated by Hb.
- the pixel numbers Wb and Hb are the resolution of the television 2 and are held in the marker information holding unit 133.
- the X coordinates X 1 and X 4 and the Y coordinates Y 1 and Y 4 in the captured image information are the coordinates M 1 (X 1 , Y 1 ) and M 4 ( X 4 , Y 4 ).
- Mi (Xi, Yi) when the detected marker is composed of a plurality of pixels, there is a problem as to which coordinate of the pixel should be adopted as Mi (Xi, Yi). Since this problem depends on the shape of the marker and the display position of the marker, it is difficult to express using a generalized mathematical expression. However, in this example, since the marker is a square and the display positions are the four corners of the display area of the display device, Mi (Xi, Yi) can be determined as follows.
- M 1 (X 1 , Y 1 ) is the coordinate of the upper left pixel in the pixel group constituting the marker 121d.
- the point information is detected by detecting the values of the coordinates M 1 (X 1 , Y 1 ) to M 4 (X 4 , Y 4 ) of the four markers 121d to 124d in the captured image information by the marker detection processing.
- Qb can be generated.
- the point information Qb is created based on the two coordinates M 1 (X 1 , Y 1 ) and M 4 (X 4 , Y 4 ).
- the point information Qb can also be created based on 1 (X 1 , Y 1 ) to M 4 (X 4 , Y 4 ). It is easier to create the point information Qb based on the two coordinates M 1 and M 4 , but the accuracy of coordinate conversion is improved by creating the point information Qb based on the four coordinates M 1 to M 4. .
- the coordinates M i are calculated in consideration of the marker movement amount in addition to the marker detection position and the like.
- the coordinates M 1 to M 4 may be calculated based on the marker detection position and detection size.
- the coordinate Qa for instructing “tone” adjustment of the operation menu 28 for adjusting image quality is designated by the user.
- the designated coordinate Qa is subjected to coordinate conversion by the display device information generation unit 137, and point information Qb is generated.
- the generated point information Qb (Xb, Yb) is transmitted to the television 2 or the like as operation information in the aforementioned step S5.
- the television 2 or the like that has received the operation information accepts the point information Qb in the operation input unit 25 as an operation for designating the coordinates Qb on the display screen of the television 2.
- “hue” adjustment (operation content display image 28 c) of the operation menu 28 has been selected, a submenu for performing hue adjustment is displayed. By continuing such an operation, a remote operation for adjusting the hue is performed.
- a part of the display device information generation unit 137 that performs a process of converting the captured image coordinate position Qa to the display area coordinate position Qb (a part that generates the point information Qb) is a “coordinate conversion unit. Is comprised.
- FIG. 22 shows a situation where the shield 4 is placed in front of the television 2 by a person other than the user, for example. Therefore, among the markers 271 c to 274 c displayed on the television 2, the marker 274 c is in a state where it cannot be photographed by the shield 4. Since the shielded marker 274c has been detected before the shield 4 is placed, it is determined that the marker is not detectable. Therefore, as described in the second embodiment, the display size of the marker 274c is increased. However, even if the display size of the marker 274c is increased, the marker 274c is not detected due to the presence of the shield 4. This is the situation.
- the shielding object 4 is not always placed near the television 2, and a cup or the like placed at the user's hand may become a shielding object, which may hinder the shooting of the marker 274 c and the like.
- the marker shielding etc. determination unit 136 determines the reason why the undetectable marker is undetectable for the undetectable marker described in detail in the second embodiment. In other words, whether or not the undetectable marker is caused by the fact that the marker cannot be photographed by some kind of shielding object, or the marker is out of the photographing range (hereinafter referred to as “shielded state”). Judge whether or not.
- the requirements for determining that the undetectable marker is undetectable due to a state such as shielding are the following two points.
- Requirement 1 is that the marker cannot be detected even if the display size change described in the second embodiment is performed a set number of times (E times).
- an undetectable counter that holds the number of times the marker is detected but not detected is introduced in the marker information holding unit 133 (FIG. 23).
- Requirement 2 is that another marker displayed on the same display device as the undetectable marker is detected.
- another marker displayed on the same display device as the undetectable marker is a detection of a plurality of markers when a plurality of markers are displayed on the display device corresponding to the undetectable marker. This refers to all markers other than the disabled marker.
- the entire display device is connected to the camera unit 11. Since it is considered that the image is out of the shooting range, it is determined that the camera is not in a blocked state.
- the marker 274c is a non-detectable marker
- the display device corresponding to the non-detectable marker 274c is the television 2.
- the markers 271c to 273c corresponding to the television 2 are “other markers”.
- the number of markers displayed by the display device is one, for example, it is possible to determine the state of shielding or the like based on the requirement 1 only. Further, for example, in this case, a configuration may be adopted in which the number of markers displayed by the display device is dynamically increased so that the state can be determined according to requirement 2.
- the non-detectable marker control unit 135 includes a case where the non-detectable marker determining unit 136 determines that the non-detectable marker is in a shielded state or the like. A function for changing the display position of the undetectable marker has been added.
- FIG. 23 shows the marker information held in the marker information holding unit 133.
- the marker information holding unit 133 includes a display instruction position that stores a marker display instruction position, a non-detectable counter, a shielding flag, a movement counter, and a marker that stores a movement amount of the marker.
- a storage area for holding the movement amount is provided (FIG. 23).
- the display instruction position (Xr, Yr) indicates that the marker is displayed such that the upper left corner of the marker is positioned at the coordinates (Xr, Yr) of the display area of the television 2 or the like.
- the non-detectable counter is used for detection of a state such as shielding described later.
- the shielding flag is a flag indicating whether or not a state such as shielding is present.
- the movement counter is used in a process of moving a non-detectable marker (a state marker such as a shield).
- the marker movement amount (Xm, Ym) indicates that the marker is displayed at a position moved from the display instruction position by Xm pixels in the X-axis direction and Ym pixels in the Y-axis direction. It is assumed that the coordinates of the display area of the television 2 and the like have the origin (0, 0) at the upper left, the X coordinate increases toward the right side, and the Y coordinate increases toward the lower side.
- the television 2 in FIG. 22 displays markers 271c to 274c at the four corners of the screen in response to a marker display instruction from the small information terminal 1 (based on the marker information described in FIG. 23).
- the marker 274 c is shielded by the shield 4.
- the portion corresponding to the marker 274c does not appear in the photographed image information generated by the small information terminal 1 (does not appear in the display unit 12 that displays the photographed image as it is). Therefore, the point information Qb cannot be calculated by detecting the above-described four corner markers.
- it is possible to calculate the point information Qb by detecting the two markers 272c and 273c for the convenience of explanation, it is assumed that re-detection of the marker 274c is attempted.
- the marker display control device mounted on the small information terminal 600 determines that the marker 274c is shielded by the following processing procedure and adjusts the display position of the marker 274c.
- the non-detectable marker display mode change process 2 (described as non-detectable marker process 2) shown in FIG. 24 is a process added to the flowchart of the non-detectable marker process shown in FIG. In this example, it is assumed that the contents of the marker state holding unit 133 shown in FIG.
- FIG. 24 is a flowchart of the undetectable marker display mode change process 2 (hereinafter referred to as undetectable marker process 2).
- Steps S441 to S443 are as described in the second embodiment.
- step S442 If it is determined in step S442 that there is an undetectable marker, a process of detecting whether the marker is in a shielded state (hereinafter referred to as a shielded state) is performed (S445). Specifically, the undetectable marker control unit 135 requests the marker shielding determination unit 136 to determine whether or not the marker 274c is undetectable due to a blocking state. The marker shielding etc. determination unit 136 performs this determination (the process of S445) according to the process shown in the flowchart of FIG.
- step S446 If it is determined in step S446 that the state is not shielded or the like, processing for increasing the instruction size is performed (S443). On the other hand, when it is determined that the state is shielded or the like, a process of moving the marker in the shielded state or the like is performed (S447).
- step S442 If there is no undetectable marker in step S442, the shielding flag, undetectable counter, etc. are turned off (non-shielded) or reset to “0” in step S448.
- step S4451 the marker information related to the undetectable marker is acquired from the marker information holding unit 133, and the set value of the shielding flag is confirmed.
- the value of the shielding flag is “shielding (on)”
- the following shielding determination process is not performed, and the marker blocking condition determination process is terminated.
- the process proceeds to the next step S4452.
- step S4452 it is determined whether or not “another marker” corresponding to the same display device as the undetectable marker is detected. When this is not detected, it is considered that the entire display device related to the undetectable marker is out of the imaging area range of the captured image input unit 11 and it is not necessary to determine the state of shielding or the like.
- the detection flags of the markers 271c to 273c, which are “other markers” of the television 2 that displays the marker 274c are “detection (on)”. Therefore, since the “other marker” of the television 2 has been detected, the requirement 2 is satisfied, and the process proceeds to the next step S4453.
- step S4453 the value of the non-detectable counter of the marker information related to the non-detectable marker is incremented.
- the value of the marker information undetectable counter related to the marker 274c is “3”, it is updated to “4” which is an incremented value.
- step S4454 it is determined whether the marker information non-detectable counter related to the non-detectable marker is equal to or greater than a predetermined set value E.
- a predetermined set value E it is determined that there is a possibility of marker detection by enlarging the display size of the marker shown in the second embodiment, and the present shielding determination is terminated.
- the value of the set value E is 4, the value of the marker information undetectable counter related to the marker 274c is “4”, which satisfies the above condition. Therefore, the requirement 1 is satisfied, and the process proceeds to next Step S4455.
- step S4455 the value of the shielding flag of the marker information related to the undetectable marker is updated to “shielding (on)”.
- the value of the shielding flag of the marker information related to the marker 274c is updated to “shielding”, and the determination processing for determining whether or not the marker is in a shielding state is ended.
- step S4453 the flow of processing when the value of the non-detectable counter is “2 or less” in step S4453 will be described.
- the initial value of the detection impossible counter is set to “0”. For this reason, after the undetectable marker is identified (S441, S442), the value of the non-detectable counter is “3 or less” in S4453 until the number of executions of the state determination process (S445) is three times. As a result, the state determination process for marker shielding and the like ends without the shielding flag being set on.
- the instruction size increase process in step S443 is performed, and the instruction size of the undetectable marker is increased. Furthermore, the television 2 that has received the marker display instruction information increases the display size of the undetectable marker 274c. However, since the non-detectable marker 274c is not photographed and detected due to the presence of the shield 4, it is specified again as a non-detectable marker in S441 and S442.
- step S446 the undetectable marker control unit 135 appropriately selects a marker (hereinafter referred to as a shielded state marker) that is determined to be undetectable (ie, the value of the shielded flag is “shielded”) due to the shielded state in the above determination process. Move to the correct position to eliminate the undetectable state due to shielding.
- a shielded state marker a marker that is determined to be undetectable (ie, the value of the shielded flag is “shielded”) due to the shielded state in the above determination process.
- the movement process (S447) of the shielding state marker will be described with reference to the flowchart shown in FIG.
- step S4471 marker information of other markers displayed on the same display device as the undetectable marker in the shielded state is acquired.
- the display device of the marker 274c which is a shielded state marker is “TV 2”, and the markers 271c to 273c are detected as other markers. Therefore, in this step, marker information of these other markers is read out.
- step S4472 the display location of the other marker and the undetectable marker in the display area of the display device is specified based on the display instruction position of the shielded state marker and the other marker, the previous detection position, and the like.
- the display location is not detailed information such as coordinates, but is information indicating a rough position where the undetectable marker is displayed in the display area of the display device, for example.
- the display locations of the markers are “upper left”, “lower left”, “upper right”, and “lower right”.
- the display location of the marker 274c is “lower right”.
- step S4473 the amount of marker movement is calculated based on the display location specified in the previous step, the display instruction position of a non-detectable marker that has been shielded (hereinafter referred to as a shielded state marker), and the instruction size.
- a shielded state marker a non-detectable marker that has been shielded
- step S4474 as will be described in detail later, marker display instruction information is transmitted to the television 2.
- FIG. 27 shows a flowchart of the marker movement amount calculation processing (S4473).
- the shielded state marker is moved in the horizontal direction (X-axis direction), vertical direction (Y-axis direction), and diagonal direction with respect to the initial display instruction position, and the marker is redetected. Is planned. If a marker is detected after the first or second movement, the subsequent movement is not performed. On the other hand, if the marker cannot be detected by the above three movements, the marker movement process ends.
- step S44731 when the value of the movement counter is 2 or less, the process proceeds to step S44732.
- the movement counter counts the number of times the state marker such as the shield is moved.
- step S44732 one of the processes in steps S44733 to S44734 is selected according to the value of the movement counter.
- Step S44733 is executed when the movement counter is “0”, and the amount of movement in the X-axis direction is calculated. Whether to move in the plus or minus direction in the X-axis direction is determined based on the display location of the non-detectable marker in the direction of moving toward the center of the display area. In this example, since the display location of the undetectable marker is “lower right”, the movement to the left side, that is, the minus side is determined.
- the movement amount is calculated so that the center of the marker is located in the center of the display area in the X-axis direction.
- the value ( ⁇ 419) is obtained by subtracting the central coordinate (931) of the marker in the X-axis direction from the central coordinate (512) of the television 2 in the X-axis direction.
- Steps S44734 and S44735 are executed, the same processing as S44733 is performed, and the center of the marker is set in the Y-axis direction, the X-axis direction, and the Y-axis from the center coordinates (512, 384) of the display device, respectively.
- the movement amount is calculated so as to be positioned at the center of the display area in the axial direction.
- step S44736 the value of the movement counter is incremented and incremented by one. Due to the increment of the movement counter, at the next execution of this marker movement process, the process of S44734 is executed by the determination of S44732, and the marker is moved in the vertical direction. Further, at the next execution, the process of S44735 is executed. Note that after the marker is moved in the X-axis direction and detection becomes possible, the marker moving process is not executed by resetting the shielding flag to OFF in S448, and the marker is moved to the Y-axis. Cannot move in the direction.
- step S44737 the marker movement amount ( ⁇ 419, 0) of the marker information holding unit 133 is set for the undetectable marker due to shielding or the like.
- FIG. 28 shows the contents of the marker information holding unit 133 in a state where the marker movement amount is set.
- step S447308 the marker movement process is terminated. Specifically, the values of the non-detectable counter and the movement counter are set to “0”, the shielding flag is turned off, the previous detection position is not set, and the indicated size is the same as the indicated size of other markers. .
- the marker 274d is not specified as a non-detectable marker, and the re-detection process of the marker 274d is not performed.
- the shielding object 4 is moved and imaging
- the display device control process may be terminated, or the display device control process may be continued using the three markers 271d to 273d.
- the marker display instruction unit 134 sends the marker display instruction information including the marker movement amount calculated above and the display instruction position to the transmission unit 15 to transmit the marker display instruction information to the device related to the undetectable marker.
- the reception unit 21 of the television 2 that has received the marker display instruction information from the transmission unit 15 inputs the information to the marker display information control unit 22.
- the marker display information control unit 22 Upon receiving this input, the marker display information control unit 22 performs “color”, “instruction size”, “display instruction position”, and “display instruction position” based on the marker display instruction information in the marker display information update process (step S16). Update “Marker movement amount”.
- the display unit 27 uses the marker (Xr + Xm, Yr + Ym) and the size (Wr, Hr) as the marker of the color specified by “color” based on the marker display information. Is superimposed.
- the display position of the marker 274c is (420, 567), and is displayed at a position avoiding the shield 4 (FIG. 29).
- the small information terminal 600 can detect the marker photographed image 124d in the photographed image information, that is, the marker 274d.
- the marker size adjustment process (S4) shown in the first embodiment is performed on the detected marker 274d, so that the display size of the marker 274d is small enough to be detected. It is possible to adjust to
- transmitting marker display instruction information including “marker movement amount” to the television 2 corresponds to “instructing to change the marker display position”.
- the marker display instruction unit 134 adds the display instruction position of the undetectable marker and the marker movement amount to calculate the “display instruction position after movement”, and the display after the movement.
- the indicated position may be transmitted to the television 2 as a part of the marker display instruction information.
- the point information Qb is generated based on the coordinates M 1 to M 4 of the marker.
- the coordinates M 4 of the moved marker 274d is a value obtained by subtracting the marker movement amount from the actual marker 274d of coordinates. That is, according to the following equation, it is predicted that the marker before the movement is detected when it is assumed that there is no shielding object 4 based on the coordinates Mi (Xi, Yi) after the movement and the movement amount (Xm, Ym).
- the coordinates Mgi (Xgi, Ygi) to be obtained are obtained.
- (Wa, Ha) is the size (number of pixels) of the display area of the television 2 in the captured image.
- (Wb, Hb) is the actual size (resolution) of the display area of the television 2 (see Formula 4).
- the point information Qb can be easily generated by calculating the coordinates Mgi.
- a part of the display device information generation unit 137 that performs the process of calculating the coordinates Mgi (Xgi, Ygi) constitutes a “position calculation unit”.
- the marker display mode changing unit 630 includes a display marker control unit 132, a marker information holding unit 133, a marker display instruction unit 134, a non-detectable marker control unit 135, and a marker shielding determination unit 136. Consists of.
- the undetectable marker process 2 can be performed in consideration of the movement of the camera unit 11. That is, when the orientation of the small information terminal 600 is changed by the user, the marker detection position changes. Then, the amount of change in the detection position can be calculated by comparing the previous detection position with the current detection position for other markers.
- the undetectable marker Based on the change amount of the detection position and the previous detection position of the non-detectable marker, it is possible to calculate the predicted detection position when it is assumed that the non-detectable marker is detected this time. If the predicted detection position is outside the range of the display unit 12 of the small information terminal 600, the undetectable marker can be determined to be in a state outside the shooting range that is located outside the shooting range. In this case, it is possible to determine that the state is out of the imaging range without performing a process of increasing the indicated size of the marker.
- the non-detectable marker When the non-detectable marker is located outside the shooting range, the non-detectable marker can be re-detected by moving the display instruction position of the non-detectable marker to the inside of the shooting range.
- the marker display control device 613 has at least one of a state in which some markers are shielded and a state in which some markers are located outside the imaging range in a state where the marker display position is inappropriate and photography cannot be performed. In this state, the marker can be moved for redetection.
- step S4473 the marker movement amount is calculated based on the display location calculated in the previous step, the display instruction position of the shielded undetectable marker, and the instruction size. In calculating this marker movement amount, the following marker movement direction and marker fluctuation amount are defined.
- the marker movement direction is a value determined based on the “display location” of the marker, and indicates in which direction the marker is moved.
- FIG. 30 is a diagram showing a table used to determine the marker movement direction.
- the marker movement direction is determined based on the display location and the random number Rp (0 to 99).
- Rp the random number
- the marker display location of the marker 274c is “lower right”, according to the random value Rp, “upper” (when Rp is 0 to 32), “left” (when Rp is 33 to 65), And “upper left” (when Rp is 66 to 99) is selected.
- either “left”, “upper left”, or “up” may be selected with equal probability, or any one may be weighted. May be easily selected.
- the marker fluctuation amount indicates how much the marker is moved in the marker moving direction. This value is calculated based on the indicated size of the undetectable marker that is shielded. Specifically, the marker fluctuation amount V is calculated by (Equation 6) as a random value that takes a value greater than 0.0 and 1.0 or less as the random number Rv.
- the marker movement amount (Xm, Ym) is calculated by (Equation 7).
- the movement direction of the marker in this mathematical expression is expressed as a movement amount in the horizontal direction as Dx and a movement amount Dy in the vertical direction.
- the marker movement amount is ( ⁇ 192, 0).
- the contents of the marker information holding unit 133 in this state are as shown in FIG.
- the marker display position is changed to various positions. Therefore, even if it is unclear which part of the display area such as the television 2 can be photographed, the marker may be detected while the marker is repeatedly moved.
- FIG. 33 is a block diagram showing the configuration of the operating device 700 and the display device 702 (TV 2, PC 3).
- the small information terminal 700 includes a marker display control device 713.
- the marker display control device 713 according to the embodiment of the present invention is a control device that is connected to the display device 702 by the communication means (5) and performs control to display a recognition marker on the display device 702.
- the marker display control device 713 includes a marker detection unit 131 that detects the recognition marker from an image obtained by capturing the display content of the display device 702, and a display device that displays the marker based on the marker detection result.
- a marker display mode changing unit 730 is provided for instructing to change the display mode of the marker 702.
- the operating device 700 can be configured by a small information terminal, for example.
- the display device 702 can be configured by a display device such as a television or a PC.
- FIG. 33 the same components as those in FIG.
- the marker display information control unit 22 of the television 2 has a function of instructing the display unit 27 to display the marker or a function of sending marker image information to the display unit 27. You may do it.
- a plurality of markers may be displayed on the display device. Further, in the small information terminals of the first and second embodiments, the coordinate position Qa (Xa, Ya) on the display device in the captured image information is changed to the coordinate position Qb (Xb, Yb) in the display area of the actual display device. A coordinate conversion function for conversion may be provided.
- the marker image included in the captured image information is detected.
- This marker detection process may be simply expressed as a process for detecting an image of a marker included in a captured image.
- the marker display control device of each of the above embodiments is typically realized as an LSI that is an integrated circuit.
- the marker display control device may be implemented as a single chip or a plurality of chips. Further, a part of the marker display control device may be made into a chip.
- LSI is used, but it may be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.
- the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible.
- An FPGA Field Programmable Gate Array
- a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.
- integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. For example, there is a possibility of adaptation of biotechnology.
- the marker display control program At least a part of the operation procedure of the marker display control device described in each of the above embodiments is described in the marker display control program.
- a CPU Central Processing Unit
- the above program may be stored in a recording medium (for example, a non-temporary recording medium) and distributed.
- the marker display control device has, for example, a function of adjusting a recognition marker to a detectable size, and is useful for a remote control device that remotely operates a display device such as a television.
- the present invention can be applied to applications such as providing a remote control function to an information terminal such as a mobile phone, a small music player, or a portable information terminal, or enhancing the remote control function of the information terminal.
- Captured image input unit (camera unit) 111 Captured Image 12 Display Unit 121 Marker 1 in Captured Image 122 Marker 2 in the captured image 123 Marker 3 in the captured image 124 Marker 4 in the captured image DESCRIPTION OF SYMBOLS 13 Marker display control apparatus 130 Marker display mode change part 131 Marker detection part 132 Display marker control part 133 Marker information holding part 134 Marker display instruction
- judgment part 137 Display apparatus information generation part 14 Equipment Operation control unit 15 Transmission unit 2 Television (display device) 21 receiving unit 22 marker display information receiving unit 23 marker display information holding unit 24 operation information receiving unit 25 operation input unit 26 content unit 27 display unit 271 marker 1 272 Marker 2 273 Marker 3 274 Marker 4 3 PC (display device) 4 Shield 5 Communication means
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Controls And Circuits For Display Device (AREA)
- Details Of Television Systems (AREA)
Abstract
Description
以下に、本発明が取り得る各種の態様を例示し、説明する。
本発明の実施の形態1に係るマーカ表示制御装置、本マーカ表示制御装置を組み込んだ操作機器、及び表示機器について説明する。
まず、操作機器たる小型情報端末1の構成について説明する。
次に、表示機器たるテレビ2の構成要素について説明する。
次に、本発明の実施の形態1の具体的使用例(図2)における処理の流れについて、図3、図4、図5、及び図6に示すフローチャートを用いて説明する。
小型情報端末1で行われる表示機器制御処理(ステップS1~S6)について説明する。なお、ステップS3,S4については、簡単に説明するにとどめ、テレビ2の処理の説明の後に詳細に説明することとする。
次にテレビ2側の処理について説明する。
次に、図5のフローチャートを用い、ステップS3のマーカ検出処理の詳細について説明する。ステップS3では、マーカ検出部131は、撮影画像入力部11が生成した撮影画像情報を受け取り、本撮影画像情報に含まれるマーカを検出する。なお、以下の説明において、上記ステップS2にて撮影画像入力部11により生成された撮影画像情報は、図8に示す撮影画像111の情報であるものとする。
図6は、ステップS4のマーカサイズ調整処理を示すフローチャートである。この図に基づいて、マーカサイズ調整処理について説明する。
以上で説明したとおり、本実施の形態1によれば、図2のようにテレビ2に大きく表示されていたマーカ271aを、図12のように、小型情報端末1が認識可能であって可能な限り小さいサイズのマーカ271aへ変更することができる。
本実施の形態において、マーカの形状は四角形に限られず、例えば、三角形、円、L字形状(鉤型)、帯状等とすることができる。特に、マーカ形状を直角三角形や、L字形状とした場合には、直角の角を表示機器の表示画面の隅に配置し、表示画面を有効に活用し得る。
上記実施の形態において、マーカの色は予め設定された色が指定されていた。
図13に、撮影画像情報における不使用色を探索する不使用色探索処理のフローチャートを示す。
ここで、マーカ検出処理においてマーカの検出数がマーカの表示数よりも多い場合に、マーカ検出処理(S3)がどのように行われるかについて説明する。なお、実施の形態1のマーカ検出処理を「マーカ検出処理1」と記載し、本変形例のマーカ検出処理を「マーカ検出処理2」と記載して区別することとする。
上記実施の形態1において、以下の処理を行うことができる。
上記実施の形態において、マーカ検出処理で輝度差合計値C(i)が算出されていた。この輝度差合計値C(i)の算出に代えて、各基準色の輝度条件を満たすか否かによって、各ピクセルが基準色と同じ色であるか否かを判別することもできる。
本実施の形態は、例えば、小型情報端末とテレビ等との距離が遠くなり、検出されていたマーカが検出されなくなった状況において、マーカの表示サイズを大きくして再検出を図るものである。
なお、上記実施の形態2において、マーカが一旦検出された後に、表示サイズを段階的に小さくしながらマーカ検出処理を行い、マーカが検出されなくなった場合に、マーカの表示サイズを検出可能なサイズに調節するという制御を行うこともできる。
本実施の形態では、マーカが障害物に遮蔽される等、マーカの表示位置が不適切であるためにマーカが撮影できなくなった場合に、マーカの表示位置を変えることで、つまり、マーカを移動することでマーカの再検出が図られる。
(1)概要
まず、本実施の形態における遠隔操作方法について説明する。
表示機器情報生成部137は、前記ステップS5において、撮影画像情報における表示機器上の座標位置Qa(Xa,Ya)(以下、撮影画像座標位置Qaという場合がある)を、実際の表示機器の表示領域における座標位置Qb(Xb,Yb)(以下、表示領域座標位置Qbという場合がある)に変換する座標変換機能を備えている。
図21の例では、小型情報端末600において、ユーザにより画質調整を行う操作メニュー28の「色合い」調整を指示する座標Qaが指定されている。その指定座標Qaが表示機器情報生成部137によって座標変換され、ポイント情報Qbが生成される。生成されたポイント情報Qb(Xb,Yb)は、前述のステップS5において操作情報としてテレビ2等に送信される。その操作情報を受信したテレビ2等は、操作入力部25においてポイント情報Qbを、テレビ2の表示画面における座標Qbを指定する操作として受け付ける。その結果、操作メニュー28の「色合い」調整(操作内容表示画像28c)が選択されたとして、色合い調整を行うサブメニューが表示される。このような操作を続けることで、色合い調整を行う遠隔操作が行われる。
次に、検出不可マーカの表示位置が不適切である場合に、検出不可マーカを移動させる処理について説明する。
マーカ遮蔽等判断部136は、実施の形態2にて詳細説明をした検出不可マーカについて、当該検出不可マーカが検出不可となっている理由を判断する。つまり、当該検出不可マーカが、何らかの遮蔽物によりマーカの撮影が不可となっている、若しくは当該マーカが、撮影範囲外であること等(以下、「遮蔽等状態」と記載する)によって生じているか否かの判断を行なう。
次に図22の構成における小型情報端末1の処理手順について、図24、図25、及び図26に示すフローチャートを用いて説明する。
ここで、図25に示すマーカ遮蔽等状態判断処理(S445)について説明する。
図24に戻り、図25のステップS4455にて遮蔽フラグがオンにされた場合、S447の処理が実行される。
図27に、マーカ移動量算出処理(S4473)のフローチャートを示す。
本実施の形態において、前述のように、マーカの座標M1ないしM4に基づいてポイント情報Qbが生成される。ここで、移動されたマーカ274dの座標M4は、実際のマーカ274dの座標からマーカ移動量を差し引いた値とされる。すなわち、次式により、移動後の座標Mi(Xi,Yi)と移動量(Xm,Ym)とに基づいて、遮蔽物4がないと仮定した場合に、移動前のマーカが検出されると予測される座標Mgi(Xgi,Ygi)が求められる。
上記検出不可マーカ処理2により、遮蔽等状態のマーカを移動させて、再検出を図ることができる。そのため、通常時は、マーカをテレビ2の表示領域の四隅に小さいサイズで表示しておくことでテレビ2の見づらさを低減しつつ、遮蔽等状態のマーカを移動させてマーカ検出を容易にすることができる。また、移動後のマーカが検出された場合、前述のステップS42においてマーカの指示サイズが小さくされ、テレビ2におけるマーカの表示サイズが小さくされる。
上記実施の形態3において、カメラ部11の動きを考慮して検出不可マーカ処理2を行うこともできる。つまり、ユーザによって小型情報端末600の向きが変えられると、マーカの検出位置が変化する。そして、他のマーカについて、前回検出位置と今回の検出位置とを比較すれば、検出位置の変化量を算出することができる。
上記実施の形態3において、遮蔽等状態のマーカは、表示領域の中央側に移動させられることが予め定められていた。それに対して、移動量をランダムに決定することもできる。
(1)前記実施の形態および変形例の説明は本発明の例示にすぎず、その範囲を限定するものではない。また、本発明の範囲を逸脱することなく種々の改良や変形を行うことができることは言うまでもない。
10 操作入力部
11 撮影画像入力部(カメラ部)
111 撮影画像
12 表示部
121 撮影画像中のマーカ1
122 撮影画像中のマーカ2
123 撮影画像中のマーカ3
124 撮影画像中のマーカ4
13 マーカ表示制御装置
130 マーカ表示態様変更部
131 マーカ検出部
132 表示マーカ制御部
133 マーカ情報保持部
134 マーカ表示指示部
135 検出不可マーカ制御部
136 マーカ遮蔽等判断部
137 表示機器情報生成部
14 機器操作制御部
15 送信部
2 テレビ(表示機器)
21 受信部
22 マーカ表示情報受信部
23 マーカ表示情報保持部
24 操作情報受信部
25 操作入力部
26 コンテンツ部
27 表示部
271 マーカ1
272 マーカ2
273 マーカ3
274 マーカ4
3 PC(表示機器)
4 遮蔽物
5 通信手段
Claims (15)
- 通信手段によって表示装置に接続され、当該表示装置に認識用のマーカを表示させる制御を行う制御装置であって、
表示装置の表示内容を撮影した画像から前記認識用のマーカを検出するマーカ検出部と、
マーカ検出の結果に基づいて、マーカを表示している表示装置に対して、マーカの表示態様を変更する指示を行うマーカ表示態様変更部と
を備えたことを特徴とするマーカ表示制御装置。 - 前記マーカ表示態様変更部は、前記マーカ検出部によって検出されたマーカを表示している表示装置に対して、マーカの表示面積を小さくする指示を行う
請求項1に記載のマーカ表示制御装置。 - さらに、マーカを繰り返し撮影する撮影画像入力部を備え、
前記マーカ検出部は、前記撮影画像入力部によって順次得られた画像からマーカを検出する処理を行い、
前記マーカ表示態様変更部は、前記マーカ検出部によって検出されたマーカが、その後のマーカ検出処理において検出されなくなった場合に、前記マーカを表示している表示装置に対して、当該マーカの表示サイズを大きくする指示、又は当該マーカの表示位置を変更する指示を行う
請求項2に記載のマーカ表示制御装置。 - さらに、表示装置に割当てたマーカの色情報を含むマーカ識別情報と、当該マーカに対応付けられた表示装置と、表示装置が表示しているマーカの表示サイズと、を少なくとも構成要素とするマーカ情報をマーカ毎に記憶するマーカ情報保持部を備えた
請求項3に記載のマーカ表示制御装置。 - 表示装置の表示領域には、相隔たる複数箇所にそれぞれマーカが表示されており、
前記マーカ表示態様変更部は、
表示装置に表示されたマーカが遮蔽物によって遮蔽されたことにより、若しくは撮影範囲からマーカが外れたことにより、当該マーカがマーカ検出部によって検出できない遮蔽等状態であると判断するマーカ遮蔽等状態判断部を備え、
前記マーカ遮蔽等状態判断部によって遮蔽等状態と判断されたマーカの表示位置を変更する指示を行う
請求項4に記載のマーカ表示制御装置。 - 前記マーカ表示態様変更部は、
遮蔽等状態ではないマーカの表示位置を、表示装置の表示領域の4隅のうちのいずれかにする指示を行い、
遮蔽等状態のマーカの表示位置を、表示領域の水平方向及び垂直方向の少なくとも一方の中央側に変更する指示を行うものである
請求項5に記載のマーカ表示制御装置。 - 前記マーカ表示態様変更部は、表示装置の表示内容を撮影した画像におけるマーカのサイズが、前記マーカ検出部にて検出したマーカの検出サイズ未満、かつ、前記画像においてマーカとして検出される下限サイズ以上になる表示サイズでマーカを表示する指示を表示装置に行う
請求項2に記載のマーカ表示制御装置。 - 前記マーカ表示態様変更部は、検出されたマーカが検出されなくなるまで、表示装置に対してマーカの表示サイズを小さくする指示を行い、マーカが検出されなくなった場合に、マーカの表示サイズを大きくする指示を行うものである
請求項3に記載のマーカ表示制御装置。 - 前記マーカ表示態様変更部は、表示装置に表示させるマーカの色を指定し、
前記マーカ検出部は、表示装置の表示内容を撮影した画像において、前記指定色のピクセルからなるピクセル群で下限サイズ以上のものをマーカとして検出するものである
請求項2に記載のマーカ表示制御装置。 - さらに、表示装置の表示内容を撮影した画像における座標を、表示装置の表示領域における座標に変換する座標変換部を備え、
前記マーカ検出部は、少なくとも前記画像におけるマーカの位置を検出し、
前記座標変換部は、少なくとも前記マーカの位置に基づいて前記座標の変換を行うものである
請求項1ないし9のいずれか1項に記載のマーカ表示制御装置。 - 前記表示装置は、表示領域にマーカと操作メニューとを含む表示内容を表示する機能を有しており、
前記座標変換部は、前記画像における表示装置の表示領域上の座標が指定された場合に、その座標を表示装置の表示領域における座標に変換するものである
請求項10に記載のマーカ表示制御装置。 - 当該マーカ表示制御装置は、前記表示装置の表示内容が撮影された画像を表示する表示部と、表示部の任意の座標を指定する操作入力が行われる操作入力部とを備えた操作機器に搭載され、
前記表示装置は、表示領域にマーカと操作メニューとを含む表示内容を表示する機能を有しており、
前記座標変換部は、操作入力部に入力された表示部の座標を、表示装置の表示領域における座標に変換するものである
請求項10に記載のマーカ表示制御装置。 - さらに、表示装置の表示内容を撮影した画像における座標を、表示装置の表示領域における座標に変換する座標変換部と、
前記遮蔽等状態のマーカの表示位置の変更量と、前記画像における前記マーカの変更後の位置とに基づいて、前記画像における前記マーカの変更前の位置を算出する位置算出部とを備え、
前記マーカ検出部は、前記画像におけるマーカの位置を検出するものであり、
前記座標変換部は、遮蔽等状態ではないマーカの位置と、前記位置算出部によって算出された遮蔽等状態のマーカの変更前の位置とに基づいて前記座標の変換を行う
請求項5又は6に記載のマーカ表示制御装置。 - 通信手段によって表示装置に接続され、当該表示装置に認識用のマーカを表示させる制御を行う集積回路であって、
表示装置の表示内容を撮影した画像から前記認識用のマーカを検出するマーカ検出部と、
マーカの検出結果に基づいて、マーカを表示している表示装置に対して、マーカの表示態様を変更する指示を行うマーカ表示態様変更部と
を備えたことを特徴とする集積回路。 - 表示装置に認識用のマーカを表示させる制御を行う方法であって、
表示装置の表示内容を撮影した画像を得るステップと、
前記画像から前記認識用のマーカを検出するステップと、
マーカの検出結果に基づいて、マーカを表示している表示装置に対して、マーカの表示態様を変更する指示を行うステップと
を備えたことを特徴とするマーカ表示制御方法。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2011521829A JPWO2011004612A1 (ja) | 2009-07-10 | 2010-07-09 | マーカ表示制御装置、集積回路、及び、マーカ表示制御方法 |
CN2010800055165A CN102292978A (zh) | 2009-07-10 | 2010-07-09 | 标识显示控制装置、集成电路及标识显示控制方法 |
US13/063,359 US20110187731A1 (en) | 2009-07-10 | 2010-07-09 | Marker display control device, integrated circuit, and marker display control method |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2009163433 | 2009-07-10 | ||
JP2009-163433 | 2009-07-10 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2011004612A1 true WO2011004612A1 (ja) | 2011-01-13 |
Family
ID=43429039
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2010/004476 WO2011004612A1 (ja) | 2009-07-10 | 2010-07-09 | マーカ表示制御装置、集積回路、及び、マーカ表示制御方法 |
Country Status (4)
Country | Link |
---|---|
US (1) | US20110187731A1 (ja) |
JP (1) | JPWO2011004612A1 (ja) |
CN (1) | CN102292978A (ja) |
WO (1) | WO2011004612A1 (ja) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2012163853A (ja) * | 2011-02-08 | 2012-08-30 | Ns Solutions Corp | 表示装置、情報提供システム、情報提供方法、及びプログラム |
JP2012233962A (ja) * | 2011-04-28 | 2012-11-29 | Konica Minolta Holdings Inc | 投影システム、投影装置、投影方法、および制御プログラム |
JP2013041348A (ja) * | 2011-08-12 | 2013-02-28 | Kyocera Corp | 携帯端末、補助情報表示プログラムおよび補助情報表示方法 |
JP2014211643A (ja) * | 2014-06-11 | 2014-11-13 | コニカミノルタ株式会社 | 投影システム、投影装置、投影方法、および制御プログラム |
JP2015126352A (ja) * | 2013-12-26 | 2015-07-06 | セイコーエプソン株式会社 | 頭部装着型表示装置、画像表示システム、および、頭部装着型表示装置の制御方法 |
JPWO2015189973A1 (ja) * | 2014-06-13 | 2017-04-20 | 三菱電機株式会社 | 情報処理装置、情報重畳画像表示装置、マーカー表示プログラム、情報重畳画像表示プログラム、マーカー表示方法および情報重畳画像表示方法 |
JP2017211766A (ja) * | 2016-05-24 | 2017-11-30 | 株式会社Screenホールディングス | 画像処理システム、画像処理プログラム、および画像処理方法 |
Families Citing this family (35)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5988563B2 (ja) * | 2011-10-25 | 2016-09-07 | キヤノン株式会社 | 画像処理装置と画像処理装置の制御方法およびプログラムと、情報処理装置と情報処理装置の制御方法およびプログラム |
US9337926B2 (en) * | 2011-10-31 | 2016-05-10 | Nokia Technologies Oy | Apparatus and method for providing dynamic fiducial markers for devices |
JP5349623B2 (ja) * | 2012-01-06 | 2013-11-20 | 株式会社東芝 | 電子機器 |
JP6014137B2 (ja) * | 2012-06-29 | 2016-10-25 | 日立マクセル株式会社 | 表示システムおよび表示端末 |
EP2904480A4 (en) * | 2012-10-04 | 2016-06-22 | Intel Corp | METHOD, DEVICE AND SYSTEM FOR MANAGING A USER-INTERFACE INTERFACE |
KR101650833B1 (ko) | 2013-04-02 | 2016-08-24 | 쓰리엠 이노베이티브 프로퍼티즈 컴파니 | 노트 인식을 위한 시스템 및 방법 |
JP6441895B2 (ja) | 2013-04-02 | 2018-12-19 | スリーエム イノベイティブ プロパティズ カンパニー | メモを管理するためのシステム及び方法 |
US10127196B2 (en) | 2013-04-02 | 2018-11-13 | 3M Innovative Properties Company | Systems and methods for managing notes |
WO2015006273A1 (en) | 2013-07-09 | 2015-01-15 | 3M Innovative Properties Company | Systems and methods for note content extraction and management by segmenting notes |
WO2015057778A1 (en) | 2013-10-16 | 2015-04-23 | 3M Innovative Properties Company | Note recognition and association based on grouping |
US10175845B2 (en) | 2013-10-16 | 2019-01-08 | 3M Innovative Properties Company | Organizing digital notes on a user interface |
US9310983B2 (en) | 2013-10-16 | 2016-04-12 | 3M Innovative Properties Company | Adding, deleting digital notes from a group of digital notes |
WO2015057793A1 (en) | 2013-10-16 | 2015-04-23 | 3M Innovative Properties Company | Multi-color channel detection for note recognition and management |
US9274693B2 (en) * | 2013-10-16 | 2016-03-01 | 3M Innovative Properties Company | Editing digital notes representing physical notes |
EP3058509A4 (en) | 2013-10-16 | 2017-08-09 | 3M Innovative Properties Company | Note recognition for overlapping physical notes |
WO2015116803A1 (en) | 2014-01-31 | 2015-08-06 | 3M Innovative Properties Company | Note capture and recognition with manual assist |
EP3159829A1 (en) * | 2015-10-21 | 2017-04-26 | Thomson Licensing | Methods of detecting and managing a fiducial marker displayed on a display device |
EP3550825A4 (en) * | 2016-12-01 | 2020-05-13 | LG Electronics Inc. -1- | IMAGE DISPLAY DEVICE AND THIS IMAGE DISPLAY SYSTEM |
US10096133B1 (en) | 2017-03-31 | 2018-10-09 | Electronic Arts Inc. | Blendshape compression system |
EP3416092A1 (en) * | 2017-06-16 | 2018-12-19 | Thomson Licensing | Method and devices to optimize marker management for pose estimation |
US10878540B1 (en) * | 2017-08-15 | 2020-12-29 | Electronic Arts Inc. | Contrast ratio detection and rendering system |
US10535174B1 (en) | 2017-09-14 | 2020-01-14 | Electronic Arts Inc. | Particle-based inverse kinematic rendering system |
US10860838B1 (en) | 2018-01-16 | 2020-12-08 | Electronic Arts Inc. | Universal facial expression translation and character rendering system |
KR102328121B1 (ko) * | 2018-02-12 | 2021-11-17 | 삼성전자주식회사 | 전자 장치 및 그 동작 방법 |
KR102338900B1 (ko) | 2018-02-13 | 2021-12-13 | 삼성전자주식회사 | 전자 장치 및 그 동작 방법 |
US10902618B2 (en) | 2019-06-14 | 2021-01-26 | Electronic Arts Inc. | Universal body movement translation and character rendering system |
WO2021087826A1 (en) * | 2019-11-06 | 2021-05-14 | Qualcomm Incorporated | Methods and apparatus to improve image data transfer efficiency for portable devices |
US11972353B2 (en) | 2020-01-22 | 2024-04-30 | Electronic Arts Inc. | Character controllers using motion variational autoencoders (MVAEs) |
US11504625B2 (en) | 2020-02-14 | 2022-11-22 | Electronic Arts Inc. | Color blindness diagnostic system |
US11648480B2 (en) | 2020-04-06 | 2023-05-16 | Electronic Arts Inc. | Enhanced pose generation based on generative modeling |
US11232621B2 (en) | 2020-04-06 | 2022-01-25 | Electronic Arts Inc. | Enhanced animation generation based on conditional modeling |
US11830121B1 (en) | 2021-01-26 | 2023-11-28 | Electronic Arts Inc. | Neural animation layering for synthesizing martial arts movements |
US11887232B2 (en) | 2021-06-10 | 2024-01-30 | Electronic Arts Inc. | Enhanced system for generation of facial models and animation |
US11670030B2 (en) | 2021-07-01 | 2023-06-06 | Electronic Arts Inc. | Enhanced animation generation based on video with local phase |
US11562523B1 (en) | 2021-08-02 | 2023-01-24 | Electronic Arts Inc. | Enhanced animation generation based on motion matching using local bone phases |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2002218565A (ja) * | 2001-01-23 | 2002-08-02 | Yamaha Corp | リモコン送信機およびリモコン送信システム |
JP2004127272A (ja) * | 2002-09-10 | 2004-04-22 | Sony Corp | 情報処理装置および方法、記録媒体、並びにプログラム |
JP2005347959A (ja) * | 2004-06-01 | 2005-12-15 | Sony Corp | 表示装置、受光装置、通信システムおよび通信方法 |
JP2008176770A (ja) * | 2006-12-21 | 2008-07-31 | Canon Inc | 帳票編集装置及び帳票編集の制御方法、プログラム |
JP2009271552A (ja) * | 2007-12-04 | 2009-11-19 | A T Communications Co Ltd | 二次元コード表示システム、二次元コード表示方法、及びプログラム |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2001142845A (ja) * | 1999-11-17 | 2001-05-25 | Toshiba Corp | コンピュータシステムおよびデータ転送制御方法 |
US7130584B2 (en) * | 2003-03-07 | 2006-10-31 | Nokia Corporation | Method and device for identifying and pairing Bluetooth devices |
JP2006330027A (ja) * | 2005-05-23 | 2006-12-07 | Matsushita Electric Ind Co Ltd | 画像出力装置、画像出力システム、およびプログラム |
JP4997783B2 (ja) * | 2006-02-15 | 2012-08-08 | 富士ゼロックス株式会社 | 電子会議システム、電子会議支援プログラム、電子会議制御装置、情報端末機器、電子会議支援方法 |
US7900838B2 (en) * | 2006-12-21 | 2011-03-08 | Canon Kabushiki Kaisha | Form editing apparatus, control method, and storage medium |
JP2008181198A (ja) * | 2007-01-23 | 2008-08-07 | Funai Electric Co Ltd | 画像表示システム |
JP5361159B2 (ja) * | 2007-09-06 | 2013-12-04 | キヤノン株式会社 | 画像表示制御装置、その制御方法、プログラム、および記録媒体 |
-
2010
- 2010-07-09 CN CN2010800055165A patent/CN102292978A/zh active Pending
- 2010-07-09 US US13/063,359 patent/US20110187731A1/en not_active Abandoned
- 2010-07-09 JP JP2011521829A patent/JPWO2011004612A1/ja not_active Withdrawn
- 2010-07-09 WO PCT/JP2010/004476 patent/WO2011004612A1/ja active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2002218565A (ja) * | 2001-01-23 | 2002-08-02 | Yamaha Corp | リモコン送信機およびリモコン送信システム |
JP2004127272A (ja) * | 2002-09-10 | 2004-04-22 | Sony Corp | 情報処理装置および方法、記録媒体、並びにプログラム |
JP2005347959A (ja) * | 2004-06-01 | 2005-12-15 | Sony Corp | 表示装置、受光装置、通信システムおよび通信方法 |
JP2008176770A (ja) * | 2006-12-21 | 2008-07-31 | Canon Inc | 帳票編集装置及び帳票編集の制御方法、プログラム |
JP2009271552A (ja) * | 2007-12-04 | 2009-11-19 | A T Communications Co Ltd | 二次元コード表示システム、二次元コード表示方法、及びプログラム |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2012163853A (ja) * | 2011-02-08 | 2012-08-30 | Ns Solutions Corp | 表示装置、情報提供システム、情報提供方法、及びプログラム |
JP2012233962A (ja) * | 2011-04-28 | 2012-11-29 | Konica Minolta Holdings Inc | 投影システム、投影装置、投影方法、および制御プログラム |
JP2013041348A (ja) * | 2011-08-12 | 2013-02-28 | Kyocera Corp | 携帯端末、補助情報表示プログラムおよび補助情報表示方法 |
JP2015126352A (ja) * | 2013-12-26 | 2015-07-06 | セイコーエプソン株式会社 | 頭部装着型表示装置、画像表示システム、および、頭部装着型表示装置の制御方法 |
JP2014211643A (ja) * | 2014-06-11 | 2014-11-13 | コニカミノルタ株式会社 | 投影システム、投影装置、投影方法、および制御プログラム |
JPWO2015189973A1 (ja) * | 2014-06-13 | 2017-04-20 | 三菱電機株式会社 | 情報処理装置、情報重畳画像表示装置、マーカー表示プログラム、情報重畳画像表示プログラム、マーカー表示方法および情報重畳画像表示方法 |
US10229543B2 (en) | 2014-06-13 | 2019-03-12 | Mitsubishi Electric Corporation | Information processing device, information superimposed image display device, non-transitory computer readable medium recorded with marker display program, non-transitory computer readable medium recorded with information superimposed image display program, marker display method, and information-superimposed image display method |
JP2017211766A (ja) * | 2016-05-24 | 2017-11-30 | 株式会社Screenホールディングス | 画像処理システム、画像処理プログラム、および画像処理方法 |
Also Published As
Publication number | Publication date |
---|---|
JPWO2011004612A1 (ja) | 2012-12-20 |
CN102292978A (zh) | 2011-12-21 |
US20110187731A1 (en) | 2011-08-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2011004612A1 (ja) | マーカ表示制御装置、集積回路、及び、マーカ表示制御方法 | |
CN106851234B (zh) | 投影仪以及投影仪的控制方法 | |
US8947488B2 (en) | Display apparatus and display method thereof | |
CN107659769B (zh) | 一种拍摄方法、第一终端及第二终端 | |
US6940558B2 (en) | Streaming content associated with a portion of a TV screen to a companion device | |
US9906762B2 (en) | Communication apparatus, method of controlling communication apparatus, non-transitory computer-readable storage medium | |
US20130265311A1 (en) | Apparatus and method for improving quality of enlarged image | |
EP4287610A1 (en) | Focusing method and apparatus, electronic device, and medium | |
CN112073788B (zh) | 视频数据处理方法、装置及显示设备 | |
CN112672062B (zh) | 一种显示设备及人像定位方法 | |
CN113645494B (zh) | 一种屏幕融合方法、显示设备、终端设备及服务器 | |
JP2009089068A (ja) | 電子機器の制御装置、制御方法及び制御プログラム | |
US10609305B2 (en) | Electronic apparatus and operating method thereof | |
CN112399220B (zh) | 一种摄像头物理开关锁定状态显示方法及显示设备 | |
CN115836528A (zh) | 一种显示设备及投屏方法 | |
CN111078926A (zh) | 一种人像缩略图像的确定方法及显示设备 | |
CN113015014A (zh) | 缩放显示方法、显示终端及存储介质 | |
CN113076031B (zh) | 显示设备、触摸定位方法及装置 | |
CN112399235B (zh) | 一种智能电视的摄像头拍照效果增强方法及显示设备 | |
KR100596140B1 (ko) | 카메라를 내장한 리모트 컨트롤 장치 및 이를 이용한포인팅 방법 | |
US11216909B2 (en) | Information processing apparatus, storage medium, and image output system | |
JPH10215388A (ja) | 遠隔操作装置 | |
CN113645502B (zh) | 一种动态调整控件的方法及显示设备 | |
WO2021248671A1 (zh) | 一种显示设备 | |
CN113242383B (zh) | 显示设备及显示设备自动对焦成像的图像校准方法 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 201080005516.5 Country of ref document: CN |
|
ENP | Entry into the national phase |
Ref document number: 2011521829 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13063359 Country of ref document: US |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 10796923 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 10796923 Country of ref document: EP Kind code of ref document: A1 |