JP2009259254A - Content expression control device, content expression control system, reference object for content expression control and content expression control program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a content expression control device for controlling one or more parameters for expression of a view movement, or the like in content without using precise and expensive position sensor and distance sensor. <P>SOLUTION: The content expression control device 200 for controlling content expression is provided with a guide image input part 243 for inputting a guide image obtained by photographing a part or whole of a reference object as a reference of controlling content expression. A control part 22 acquires correspondence relation information for acquiring correspondence relation information of the guide image to the reference object, and generates control data for controlling a parameter for content expression on the basis of the acquired correspondence relation information. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

The present invention relates to a content expression control device, and in particular, a portable content expression control device and a content expression control system for browsing content that is relatively large or has a large amount of information per page using a relatively small display screen. relates to the content representation control for the reference object body contact and content representation control program.

  Personal computers, word processors, workstations, etc. (hereinafter referred to as “PCs”) usually have a relatively large screen, so when viewing electronic documents that are relatively large or have a large amount of information per page. Is suitable.

  However, when the amount of information per page is larger, there is a case where it is desired to enlarge and display the electronic document so that it is larger than the screen. In some cases, it is desired to display a part of a large electronic document in a small window arranged on a part of the screen. In such a case, in a PC or the like, a user interface member called a scroll bar is provided at the right end or lower end of the display portion, and a visible range (view) on the electronic document is controlled through an operation with a pointing device such as a mouse. In addition, there is a method of temporarily transitioning to the scroll-only mode and performing view control by a mouse drag operation or the like. This method is suitable for a mouse or the like, but is not suitable for a pen-type input device that operates directly on the screen because it cannot be dragged outside the screen.

  In order to control the enlargement / reduction ratio of the electronic document to be displayed, a method of designating numerically, a method of selecting a predetermined enlargement ratio from a menu, a method of operating continuously with a slide bar, or an enlargement / reduction ratio There is a method (so-called magnifying glass tool) for enlarging / reducing around a predetermined portion by transitioning to a reduction mode and clicking with a pointing device.

  On the other hand, in a portable information device, it is not preferable to use a mouse or the like that is supposed to be operated on a desk, so there are four buttons for scrolling in four directions, or a display screen with a pressure-sensitive tablet attached. In some cases, the scroll bar is displayed on the screen and operated with the pen-type instrument described above. However, with such a method, especially when the electronic document is large, it is not intuitive when moving the view to an arbitrary location, which causes the operator's stress and takes a lot of operation time. End up.

  Therefore, for example, in Non-Patent Document 1, in order to realize an intuitive scrolling operation of an electronic document in an information device with a small screen, a mouse ball mechanism is incorporated on the back of the information device and the amount of movement is moved when moving on the desk. In addition, a mechanism is disclosed in which display contents are scrolled according to a moving direction.

  As a mechanism similar to Non-Patent Document 1, Non-Patent Document 2 uses a system in which a device combining a barcode reader and a pen mouse is connected to a small information device. According to this Non-Patent Document 2, a barcode is pasted at a specific position on the wall, the surface of the wall is traced with the device from the barcode position as a base point, and displayed on the display unit of the small information device according to the displacement of the device. By scrolling the information, a mechanism that makes you feel as if you are looking into the wall is disclosed.

  In addition, in the technical field called “Augmented Reality” (translation of Augmented Reality, sometimes translated as Enhanced Reality, Augmented Reality, Mixed Reality, etc.), the real world is photographed with a camera connected to a computer. An attempt has been made to superimpose and display a virtual document or image (hereinafter referred to as a virtual image) generated by a computer on a specific object in a photographed real world image. According to this, it is possible to realize a system that controls the appearance of the virtual image based on the positional relationship between the camera and the specific object.

  For example, Non-Patent Document 3 discloses a method for recognizing the position of a two-dimensional matrix code placed in the real world and controlling the display position of a virtual image associated therewith.

  Japanese Patent Application Laid-Open No. 2004-228688 measures relative position information including a distance from a camera to an identifier added to a specific object (access element), and changes display contents of a virtual image corresponding to the identifier according to the distance. Is disclosed.

  Further, Patent Document 2 captures a plurality of signs installed in the real world with a camera, recognizes the position in the screen, and adjusts the display position of the virtual image, thereby corresponding to the real-world object and the corresponding object. An apparatus that attempts to reduce the deviation from the virtual image display position is disclosed.

Japanese Patent No. 2881973 JP 2000-41173 A

Ichiro Shiio, “Scroll Display: Indicating device for micro information equipment”, Transactions of Information Processing Society of Japan, Vol.39, No.5, pp.1448-1454, May 1998, Information Processing Society of Japan ISSN 0387- 5806 (Received July 1, 1997, accepted December 1, 1997) Ichiro Shiio, “Scroll Browser: Browser in the Wall”, Interaction'99 Proceedings ISSN 1344-0640 IPSJ Symposium Series Vol.99, No.4, pp.39-40, March 1999 Junichi Kyokumoto, “Augmented Reality Construction Method Using 2D Matrix Code”, Interactive System and Software 4 ISBN4-7649-0258-3 Japan Software Science Society WISS '96, edited by Jiro Tanaka, Modern Science

  However, in the conventional technology described above, for example, Non-Patent Document 1 described above, it is assumed that the operation is mainly performed on a desk, but even when the absolute position of the target view in the electronic document is known. In order to move the view, trial and error is required.

  For example, when the view is moved to an absolute position in the electronic document, such as “the center of the lower end”, the operator must move the device at the same time while following the changing display contents. At this time, assuming a case where an operation operation is performed so as to exceed the lower end of the electronic document, two types of correspondence are mainly conceivable.

  First, when the lower end is exceeded, a blank is displayed, and at the same time, the view coordinates corresponding to the exceeded distance are maintained. However, in the first correspondence, in a state where a blank is displayed, it is difficult to determine where the current view is based on the position of the electronic document, so it is difficult to return the view.

  Second, when the lower end is exceeded, the image is stopped at the lower end. However, in the second correspondence, since the correspondence between the origin on the electronic document and the origin on the desk is lost, it is contrary to the intuition of the operator and stable operation cannot be performed.

  Further, in Non-Patent Document 1, taking a map as an example of an electronic document, “scrolling to the Nagoya part using the positions of Tokyo and Osaka” (Non-Patent Document 1, p. 3, 10 from the top left) (Line) Consider the case. In this case, if the map is large and only the position of Tokyo (first point) is visible at first, to move to Nagoya (third point), the premise is to move to the position of Osaka (second point) in advance. It is necessary to prepare for moving and confirming the physical positions of the devices on the desk corresponding to the first point and the second point. During this preparatory operation, the operator must search the second point by moving the device at the same time while following the changing display contents, which causes eye fatigue and increases the operation time. there is a possibility. In addition, since the physical position of the display part is moved in the operation on the desk, the operator is forced to track the position of the display part with a line of sight, which causes eye fatigue. there is a possibility.

  Further, Non-Patent Document 1 proposes a mouse ball operation using a hand in an environment where there is no desk such as a destination (Non-Patent Document 1, p.3, 5th line from the lower left), but it is large. If you want to move the view to a distant place in an electronic document, you will need to rotate the mouse ball over and over again, which is less intuitive and may take more time than on a desk. There is a problem of having sex.

  Furthermore, in the method of Non-Patent Document 2, when tracking is shifted in a state where the operation device leaves the barcode (origin), it is necessary to return to the barcode again. Possible causes of tracking shift include that the direction of the operation device must be constant (it is difficult to maintain constant), the case where the device is separated from the wall surface, or the case where the wall surface is uneven.

  In the inventions disclosed in Non-Patent Document 3, Patent Document 1, and Patent Document 2, in a portable system, the configuration for precisely controlling a view by approaching a specific object is not necessarily clear. .

The present invention has been made to solve such a problem, and is suitable for a portable device, and can easily control one or more parameters related to expressions such as view movement in content. controller, content presentation control system, and an object thereof is to provide a content presentation control reference product body contact and content presentation control program.

Another object of the present invention is content expression control capable of easily controlling parameters related to expression even when a camera is brought close to a specific real object corresponding to the content and the contour thereof is outside the shooting range. device, the content presentation control system is to provide a content presentation control reference product body contact and content presentation control program.

According to one aspect of the present invention, the content expression control device
A photographing means capable of photographing the subject by freely moving the device main body held by the user ;
An image display means for displaying content, which allows the user to freely move the device body held by the user and view the display surface ;
Reference image data storage means for storing reference image data for collation, in which the photographing means has photographed a reference object serving as a reference for display control of the content;
A guide image data input means for inputting data of a guide image obtained by photographing the part or the whole of the reference object by the photographing means;
The features of the input guide image and the reference image are collated, and the display range of the guide image relative to the reference image, which corresponds to which partial range of the reference image the guide image corresponds to, is specified. Image collation means for obtaining corresponding range information;
Display control data generating means for generating display control data for controlling at least a display range of a display image displayed by the image display means based on corresponding range information acquired by the image collating means;
Display control data output means for outputting display control data generated by the display control data generation means;
Display control means for controlling the image display means based on display control data output by the display control data output means,
Corresponding range information acquired by the image collating unit includes position information in the reference image of the guide image to be collated,
The display control data generated by the display control data generating unit includes position information that specifies a display range of the display image.

  Preferably, the display control data generation unit or the display control unit partially restricts a change in a parameter related to content expression with respect to a change in the guide image.

Preferably, the display control data generating means is
Distance correspondence information calculating means for calculating distance correspondence information corresponding to the distance from the photographing means to the reference object based on the acquired correspondence range information;
The display control data is generated using the distance correspondence information.

Preferably, the content expression control device is
Correspondence relationship switching means for switching the correspondence relationship between the specific display control data and the distance correspondence information is further provided.

According to another aspect of the present invention, the content expression control program is
A content expression control program for controlling content expression,
A shooting step that allows the user to freely move the device body held by the user and photograph the subject ,
An image display step for displaying content, which allows the user to freely move the device body held in his hand and view the display surface ;
A reference image data storage step of storing reference image data for collation, in which the shooting step images a reference object that is a reference for display control of the content;
A guide image data input step in which the shooting step inputs data of a guide image obtained by shooting a part or the whole of the reference object;
The features of the input guide image and the reference image are collated, and the display range of the guide image relative to the reference image, which corresponds to which partial range of the reference image the guide image corresponds to, is specified. An image matching step for obtaining corresponding range information;
A display control data generating step for generating display control data for controlling at least a display range of a display image displayed by the image display step based on corresponding range information acquired by the image collating step;
A display control data output step for outputting the display control data generated by the display control data generation step;
Causing the computer to execute a display control step for controlling the image display step based on display control data output by the display control data output step;
The corresponding range information acquired by the image matching step includes position information in the reference image of the guide image to be verified,
The display control data generated by the display control data generation step includes position information that specifies a display range of the display image.

According to yet another aspect of the present invention, the reference object for content expression control is
A reference object serving as a reference for the content expression control for a content expression control device for controlling the expression of content,
Provided with a coordinate indication marker indicating the coordinates on a plurality of coordinate systems on the surface of the surface that can be photographed from outside
The coordinate indicating marker is
A reference pattern for causing the content expression control device to recognize the coordinate indicating marker;
A coordinate information pattern representing coordinate information of the position of the reference pattern,
At least one of the reference pattern and the coordinate information pattern is colored with a specific color,
The content expression control device includes:
Photographing means;
Image display means for displaying content;
A guide image data input means for inputting data of a guide image obtained by photographing the part or the whole of the reference object by the photographing means;
In the guide image input by the guide image data input means, important marker determining means for determining the position of the guide image corresponding to the reference object and determining at least two of the coordinate indicating markers as important markers;
Correspondence information acquisition means for acquiring correspondence information specifying a position corresponding to the guide image with respect to the reference object from the important marker determined by the important marker determination means;
Display control data generating means for generating display control data for controlling at least a display range of a display image displayed by the image display means based on the correspondence information acquired by the correspondence information acquiring means;
Display control data output means for outputting display control data generated by the display control data generation means;
Display control means for controlling the image display means based on display control data output by the display control data output means.

According to yet another aspect of the present invention, a content expression control system
A content expression control system comprising: a content expression control device that controls expression of content; and a reference object that serves as a reference for the content expression control for the content expression control device ,
The reference object is provided with a coordinate indication marker indicating coordinates on a plurality of coordinate systems on a surface portion that can be photographed from the outside,
The content expression control device includes:
Photographing means;
Image display means for displaying content;
A guide image data input means for said imaging means for inputting data of a guide image shooting part or all of the reference object,
In the guide image input by the guide image data input means, important marker determining means for determining the position of the guide image corresponding to the reference object and determining at least two of the coordinate indicating markers as important markers;
Correspondence information acquisition means for acquiring correspondence information specifying a position corresponding to the guide image with respect to the reference object from the important marker determined by the important marker determination means ;
Display control data generating means for generating display control data for controlling at least a display range of a display image displayed by the image display means based on the correspondence information acquired by the correspondence information acquiring means ;
Display control data output means for outputting display control data generated by the display control data generation means;
Ru and display control means for controlling said image display means based on the display control data output from the display control data output means.

  In the present invention, content parameter control is performed following the change of the guide image, so that parameters such as the display position of the content can be controlled smoothly. In particular, when two-dimensional or more coordinate information is acquired and used as correspondence information, different parameters such as a display position and an enlargement ratio can be controlled simultaneously or continuously.

  Further, when the content expression control apparatus further includes a photographing unit, the operator can intuitively manipulate the positional relationship between the photographing unit and the reference object, photograph the reference object and use it as a guide image. And it becomes possible to control instantaneously.

  Also, the parameters can be controlled by moving only the object to be photographed without moving the content expression control device, and when the content is an image, fatigue due to movement of the operator's line of sight can be reduced.

It is a figure which shows the example of a structure of a mobile telephone as a specific example of the content expression control apparatus in this Embodiment. It is a figure which shows the example of the hardware constitutions of a mobile telephone as a specific example of the content expression control apparatus in this Embodiment. It is a figure which shows the example of the minimum hardware constitutions of the content expression control apparatus in this Embodiment. It is a block diagram which shows the function structure of the content expression control apparatus in this Embodiment . It is a figure which shows the specific example of the guide sheet in this Embodiment. It is a figure which shows the specific example of the coordinate instruction | indication marker in this Embodiment. It is a flowchart which shows the process at the time of performing the content expression control in this Embodiment. It is a figure which shows the example of the guide sheet in this Embodiment, and its imaging | photography part. It is the figure which extracted and showed only the important marker in the guide image data in this Embodiment. It is the figure which showed the example of the content (electronic map) in this Embodiment, and the view of a content. It is a figure which shows an example of the search method of a coordinate instruction | indication marker. It is a figure for demonstrating the detailed method for reading the coordinate value described in the coordinate instruction | indication marker. It is a figure for demonstrating the example which implement | achieved the coordinate instruction | indication marker using color information. It is the figure which modeled and showed a mode that the arbitrary points on a guide sheet were projected on the imaging surface in a CCD camera as one point in a guide image. It is a figure which shows the example of the image used as the origin of a guide image. It is a figure which shows the other example of the image used as the origin of a guide image. It is a figure which shows the example of the mobile telephone in which a CCD camera is not located in the center back surface of LCD. It is a figure which shows the example of the guide sheet corresponding to the mobile telephone in which a CCD camera is not located in the center back surface of LCD. It is a figure which shows the example of the guide sheet corresponding to the some mobile phone from which the attachment position of a CCD camera differs. It is side surface sectional drawing and rear view which show the example which attached the correction tool to the mobile telephone in which a CCD camera is not located in the center back surface of LCD. It is a block diagram which shows the function structure of the content expression control system which makes the control object the view of the surveillance camera which can be controlled remotely.

[Embodiment 1]
Embodiments of the present invention will be described below with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

FIG. 1 is a diagram illustrating a specific example of a content expression control apparatus according to the present embodiment.
In the present embodiment, as an example of the content expression control apparatus, description will be made assuming that the hardware of a mobile phone is used.

  1A shows the front surface of the mobile phone, and FIG. 1B shows the back surface of the mobile phone. The mobile phone includes a housing 101, an LCD (Liquid Crystal Display) 102, an operation button group 103, a CCD (Charge-Coupled Devices) camera 106, a microphone 105, and a speaker 104.

  Next, FIG. 2 shows a specific example of the hardware configuration of the content expression control apparatus according to the present embodiment.

  Referring to FIG. 2, content expression control apparatus 200 in the present embodiment is configured by a CPU (Central Processing Unit) and the like, and includes a control unit 22 that controls the entire apparatus, and data input to the outside of the apparatus. It is composed of an input / output unit 24 that performs output, a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and is received from a program executed by the control unit 22, intermediate data of the program, or another computer And a storage unit 23 for storing data and the like.

  Further, the input / output unit 24 includes an operation unit 241 including numeric buttons such as “1” and “2” and menu item instruction / selection buttons, a display unit 242 such as an LCD for displaying information to the user, and an image. An image input unit 243 such as a CCD camera for inputting a sound, a communication unit 244 for communicating with other devices, a sound input unit 245 such as a microphone for inputting sound, and a sound output unit 246 such as a speaker for outputting sound. Including.

  Note that the hardware configuration shown in FIG. 2 is a general mobile phone hardware configuration, and the hardware configuration of the content expression control device according to the present invention is limited to the hardware configuration shown in FIG. It is not something.

  For example, FIG. 3 is a diagram showing a minimum configuration of the content expression control device according to the present invention. In the content expression control device 200b shown in FIG. 3, the input / output unit 24b has parameters related to expression of the target content (position of the display portion, orientation, display magnification, playback time point on the time axis, etc. An expression control data generation unit 242b that generates data for controlling the image data) and an image data input unit 243b that inputs guide image data. The internal structure of the storage unit 23b is the same as that of 23, and is omitted. In this configuration, the LCD 102 and the CCD camera 106 that input and output images may be connected to the outside of the content expression control device 200b according to the present invention when in use.

  Next, FIG. 4 is a block diagram showing a functional configuration of the content expression control device according to the present embodiment. Referring to FIG. 4, the content expression control apparatus according to the present embodiment includes an image data acquisition unit 301 that acquires guide image data 311 in which a reference object for controlling expression is captured, and a guide image for the reference object. A correspondence information acquisition unit 302 that acquires correspondence information such as a position and an enlargement / reduction ratio, and controls parameters related to expression, such as the upper left coordinates of the content view and the enlargement ratio, based on the acquired correspondence information. An expression control data generation unit 303 that generates data for the control, and a parameter control unit 304 that controls parameters related to the expression of the input content 312 based on the expression control data and outputs the parameters as output data 313. . The function of the image data acquisition unit 301 is realized by, for example, the image input unit 243 in FIG. 2, and the other functions are performed by the control unit 22 reading a program stored in the storage unit 23 and executing the program. This is achieved by controlling each part of the hardware configuration shown in FIG. For example, according to a program stored in the storage unit 23, the control unit 22 may control to acquire image data and generate expression control data only while a specific operation button is pressed. The output data 313 is given to the display unit 242 in FIG. 2, for example.

  Next, FIG. 5 shows a specific example of a guide sheet 401 on which a pattern from which an image serving as the guide is based is printed. The guide sheet 401 is printed with a large number of coordinate indicating markers 402 indicating two-dimensional coordinates with a certain point on a plane including the surface as the origin.

  Each numerical value 403 on the guide sheet 401 is a value corresponding to the horizontal axis of each coordinate indicating marker, that is, a scale for clearly indicating the X coordinate. For example, “10” indicates an X coordinate of 10 × 256. . Similarly, each numerical value 404 on the guide sheet 401 is a scale for easily indicating the Y coordinate. These scale values 403 and 404 are given for reference, and are not necessarily required for the present embodiment.

FIG. 6 shows an example of the detailed structure of the coordinate indicating marker 402.
FIG. 6A shows an example pattern 506 of the coordinate display marker 402. The coordinate instruction marker pattern 506 includes a reference pattern 501 that serves as a reference for recognition, coordinate information patterns 504 and 505 that respectively represent the values of the X coordinate and the Y coordinate of the center position of the reference pattern, and directions in which the coordinate information patterns exist. And a direction indicator 502 indicating.

  FIG. 6B is a correspondence table between coordinate information patterns and coordinate values. With reference to FIG. 6B, the coordinate values are shown in hexadecimal. For example, “0x100” indicates a coordinate value of 100 in hexadecimal, that is, 256 in decimal, and the lower right corner of the corresponding coordinate information pattern 511 is filled. The lower right corner, lower left corner, upper right corner, and upper left corner of the coordinate information pattern are the ninth bit, the tenth bit, the eleventh bit, and the twelfth bit counted from the lower order when the coordinate value is expressed in binary. , Respectively. For example, FIG. 6A shows that the center coordinates (X, Y) of the reference pattern 501 are (0x200, 0x100). That is, returning to FIG. 5, each coordinate indicating marker (402, etc.) expresses two-dimensional coordinates (X, Y) that are different on the sheet.

  Next, processing when content browsing control is executed in the content expression control apparatus according to the present embodiment will be described with reference to FIG. 7, FIG. 8, FIG. 9, and FIG.

  FIG. 7 is a flowchart showing the flow of processing when executing content expression control according to the present embodiment.

Referring to FIG. 7, first, guide image data is acquired in step S01. For example, referring to FIG. 5, the guide image data is obtained by photographing a partial area 700 of a guide sheet 401 such as 401 with the CCD camera 106, and calculating the luminance of each point of the photographed image with the function of the image input unit 243. It is converted to the array of. In this embodiment, for simplification of explanation, the guide image data is monochrome, and the guide sheet 401 and the view 700 are rotated by the CCD camera (centered on a plurality of axes), and the distance between the center of the view and the periphery is changed. It is assumed that there is no shift due to differences, lens characteristics, sheet distortion, and the like. (Actually, correction processing may be necessary for these.)
In step S02, the brightness and contrast of the guide image data are corrected. For example, if correction is performed so that the average value and variance of the brightness of each pixel constituting the guide image data are close to a certain value, the processing in the next step may be facilitated.

  In the next step S03, the guide image data is binarized. That is, for example, the value of the luminance value of each pixel is converted to a value of 1 if it is greater than or equal to a certain threshold, and 0 if it is less than the threshold.

  Next, in step S04, the coordinate pointing marker is searched, and the coordinate data of the center position of the reference pattern (501 in FIG. 6) at the left end of the coordinate pointing marker image existing in the guide image data is listed. The coordinate data here is not coordinate information on the guide sheet 401 but coordinate information based on the upper left origin of the guide image data 700. Note that any method may be used for searching the reference pattern of the coordinate indicating marker, and an example thereof will be described later. By this step S04, most of the coordinate data of the center position of the reference pattern of the coordinate indicating marker in the partial area 700 of FIG. 8 is listed.

  However, the reference pattern to the right of the certain virtual boundary line 704 is not listed. This is because the coordinate information pattern held by the reference pattern on the right side of 704, for example, 703, is outside the screen and cannot be used in the subsequent step S05. That is, in step S04, coordinate data in the guide image data relating to 9 rows × 3 columns = 27 coordinate indicating markers including 701 is listed.

  In step S05, an important marker is selected / determined among the coordinate indicating markers listed in step S04. The important marker is a minimum coordinate indicating marker necessary for specifying the position of the guide image data 700 (that is, the upper left coordinate and the area size) with reference to the origin in the guide sheet 401. . Here, of the coordinate data listed, two of the smallest coordinate 801 and the largest coordinate 802 are determined as important markers. Here, it is assumed that there is no shift due to rotation or the like. In this case, two important markers are sufficient.

  FIG. 9 is a diagram showing only important markers in the guide image data. Referring to FIG. 9, with the upper left corner of guide image data 800 as the origin, the center coordinates of the reference pattern of marker 801 are (X1, X1), and the center coordinates of the reference pattern of marker 802 are (X2, X2). It is shown that. Next, in step S06, coordinate values (coordinate values based on the origin of the guide sheet) described in each important marker are read. When the coordinate information pattern in the important marker 801 is compared with the table of FIG. 6B, it can be seen that the coordinate value is (0x200, 0x200) in hexadecimal, that is, (512, 512) in decimal. Similarly, it can be seen that the important marker 802 has a coordinate value of (0xA00, 0x600) in hexadecimal, that is, (2560, 1536) in decimal. The detailed procedure for reading coordinate values will be described later.

  Here, FIG. 10 shows an example of content (electronic map) and content view in the present embodiment.

  In FIG. 7, in the next step S07, a display range 901 for displaying a part of the content 900 on the LCD 102 is calculated with reference to FIG. Assume that each coordinate on the content 900 (electronic map) corresponds to each coordinate on the guide sheet 401. Here, the coordinates (Xp, Yp) on the content of the upper left vertex 902 of the view 901 and the coordinates (Xq, Yq) on the content of the lower right vertex 903 are obtained. When the variables given in FIGS. 9 and 10 are used, the following is obtained.

Xp = (X2, Xa-X1, Xb) / (X2-X1)
Yp = (Y2, Ya-Y1, Yb) / (Y2-Y1)
Xq = Xp + Xv. (Xa-Xp) / X1
Yq = Yp + Yv. (Ya-Yp) / Y1
The value of the variable on the right side has been calculated in the previous steps.

  In step S08, the content view is cut out and displayed. For example, assuming that the content 900 is bitmap data, luminance data at each point on the content corresponding to each pixel of the display memory is cut out and transferred to the display memory. The display memory and the guide image data do not necessarily have the same resolution. However, for the sake of simplicity, the display memory and the guide image data correspond to, for example, the coordinates (Xw, Yw) of a certain pixel in the display memory. The coordinates (Xr, Yr) in the content to be calculated are calculated by the following formula, transferred to an address corresponding to (Xw, Yw) in the display memory, and used for display on the LCD 102.

Xr = Xw. (Xq-Xp) / Xv + Xp
Yr = Yw · (Yq−Yp) / Yv + Yp
If the content 900 is vector data, the coordinate data (Xr, Yr) of each feature point in the content is converted to the coordinates (Xw, Yw) of the corresponding pixel in the display memory by the following equation. be able to.

Xw = Xv. (Xr-Xp) / (Xq-Xp)
Yw = Yv · (Yr−Yp) / (Yq−Yp)
In the case of vector data, instead of transferring luminance data, graphic data is rendered on the display memory based on the obtained coordinate group. At this time, the portion where the figure is outside the screen is excluded by the clipping process. Finally, in step S09, the process branches depending on whether or not an end instruction is detected. If not detected, the process returns to step S01 and repeats. If detected, the process ends.

  Here, an example of the search method of the coordinate indicating marker in the above-described step S04 will be described with reference to FIGS.

  FIG. 11B represents the reference pattern 501 at the left end of the coordinate indicating marker by a two-dimensional array of numerical values. That is, the black part is represented by 0, and the white part is represented by 1. FIG. 11A is an example of data obtained by scanning each pixel of the guide image data binarized in step S03 in order from the upper left to the lower right, and acquiring a total of 5 × 5 pixels around the reference pattern. .

  In FIG. 11A, the pixel that should originally be 0 is 1 or vice versa, which is an error from the original reference pattern FIG. 11B. As described above, in the image data captured by the camera, an error often occurs due to the influence of noise, position shift, and the like.

  Here, FIG. 11C shows an array resulting from subtracting the corresponding elements in FIG. 11B from the elements in FIG.

  In FIG. 11C, the total sum of the absolute values of the elements is 3. This value can be used as an evaluation value of pattern similarity, and it can be evaluated that the smaller the evaluation value is, the closer the center of FIG. 11A is to the center of the reference pattern.

  For example, if this value calculated around a certain coordinate is below a certain threshold value, that coordinate is a candidate for the central coordinate of the reference pattern, and the coordinate whose evaluation value is minimum within a total of N × N pixels around that coordinate is selected. What is necessary is just to make it the object of the said list-up.

  In practice, since the distance between the CCD camera and the guide sheet is not constant, it is preferable that the pattern in FIG. 11B is collated a plurality of times while changing the size around one pixel in the guide image. Although FIG. 11B shows 5 × 5 pixels, for example, it is better to collate with patterns similar to similar shapes such as 7 × 7 pixels and 9 × 9 pixels. A value obtained by dividing the sum of the absolute values of the differences between the elements obtained here by the number of pixels of each pattern can be used as the evaluation value. This also makes it possible to estimate the approximate distance between the CCD camera and the guide sheet. In this embodiment, the guide image is collated with a binarized pattern. However, it is not always necessary to binarize, and the guide image may be collated using a luminance image with a gradation of luminance.

  Here, a detailed method of reading the coordinate value on the guide sheet described in the important marker in the above-described step S06 will be described with reference to FIG.

  FIG. 12 shows an example of an important marker in the guide image, where a black circle indicates a black pixel and a white circle indicates that a white pixel exists at each position.

  In step S04 described above, the coordinates in the guide image of the center point 1103 of the reference pattern have already been obtained. When each pixel is scanned in one direction starting from the center point 1103 and in the opposite direction (here, left and right directions), patterns of “black, white, and black” are detected. It is recognized that the distance from the black point 1101 to the end of each pattern to the black point 1105 is the apparent size u (shown in FIG. 12) of the reference pattern.

  When the scanning in a certain direction (right here) is further advanced, a point 1106 on a black area sandwiched between white areas on the top and bottom is found. Thereby, it is recognized that the coordinate information on the guide sheet exists in the direction 1106 with the center point 1103 as a reference.

  When scanning rightward from the point 1105, the pattern of “black, white, black” is detected again. Coordinate information exists ahead of the black dot 1108 that is the end of the pattern. The distance from the point 1101 to the point 1108 should be approximately 2u (illustrated). The closer to this value, the higher the recognition accuracy of the coordinate pointing marker, and conversely, this value deviates from a certain threshold value. It can be determined that recognition has failed.

  If it fails, it returns to one of the previous steps as error processing. For example, returning to S05, processing such as selecting another important marker may be performed.

  Note that such error processing is not the essence of the present embodiment, and is therefore omitted from the flowchart of FIG.

  Next, when the pixel is examined with a distance of about 0.3u in the direction of the lower right, lower left, upper right, and upper left centering on the point 1109 of 2.5u from the left end of the marker, the X coordinate on the guide sheet corresponding to the point 1103 Can be recognized. Similarly, when a pixel is examined centering on a point 1110 at 3.5u from the left end of the marker, a pattern representing the Y coordinate on the guide sheet corresponding to the point 1103 can be recognized. A correspondence table showing what coordinate values these patterns represent is as shown in FIG.

  As described above, in the first embodiment, at least two coordinate indicating markers (however, the X coordinate and the Y coordinate are not the same) are within the imaging range due to the positional relationship between the guide sheet and the CCD camera. If so, the content can be displayed in correspondence with the position of the guide sheet.

  In the first embodiment, as shown in the flowchart of FIG. 7, even when the positional relationship between the guide sheet and the CCD camera changes every moment, the processing is performed following the change. That is, scrolling and zooming operations can be performed smoothly. In particular, scrolling and zooming operations can be performed simultaneously or continuously. Further, by photographing a specific position of the guide sheet, the view can be moved intuitively and instantaneously. The view can also be controlled by moving only the guide sheet without moving the mobile phone device, and fatigue due to movement of the operator's line of sight can be reduced.

  Note that the virtual size of the content depends on the coordinate information printed on the guide sheet. Here, for high-accuracy view control, it is preferable to use a guide sheet whose change in coordinates corresponding to the actual length in the guide sheet is large (that is, large as a whole), and a wide range of views with few operations. In order to perform the control, it is preferable to use a guide sheet having a small change in coordinates (that is, small as a whole). In this way, a plurality of guide sheets having different coordinate changes relative to the actual length may be prepared and used, or the coordinate change may be logically changed by multiplying the coordinate information by a specific coefficient in the program. Good.

  Further, when a special guide sheet having a partially different coordinate change is used in one guide sheet, view control can be performed with different accuracy depending on the content portion. For example, if the content includes an image of a seal, the use of a guide sheet with a large coordinate scale corresponding to the position of the seal makes it easy to check the stamp. Further, a guide sheet obtained by cutting out and enlarging a specific area may be prepared separately and used as necessary.

[Variation 1 of Embodiment 1]
In the mobile phone 1, for the sake of simplicity, it is assumed that a shift due to rotation about any coordinate axis in the three-dimensional space is negligible between the CCD camera and the guide sheet.

  However, in practice, it may be difficult for the operator to hold the guide sheet and the CCD camera accurately without relatively rotating as in the first embodiment. In addition, by intentionally rotating the content, the long and short sides of the content view can be changed to a convenient orientation, and as shown in a bird's-eye view, the adjacent portion is large and wide, including far away. It is convenient if you can browse.

  Even in such a case, in principle, processing according to the flowchart of FIG. 7 is possible. However, the contents of processing in some steps described below are different.

(Processing of step S04)
First, in the case of searching and listing the coordinate indicating marker in step S04, if the marker in the guide image is extremely deformed by photographing the guide sheet from an oblique direction, pattern matching with a shape as shown in FIG. Makes it difficult to search.

  Therefore, for example, if the reference pattern of the coordinate indicating marker is colored with a specific color (red, etc.), photographed with a color-compatible CCD camera, and a guide image holding color information is obtained, the reference pattern can be easily searched. can do. In this case, in step S03, instead of binary, it is preferable to convert to three or more values such as red, white, and black.

(Process of step S05)
In step S05, four important markers are determined for reasons described later. For example, first, two markers that are most distant from each reference pattern in the guide image are determined as important markers, and then a marker that includes the reference pattern that is the most distant from the straight line connecting the two reference patterns is selected. The third important marker may be used as the fourth important marker. Finally, the fourth important marker may include a reference pattern that is farthest from the reference pattern of the third important marker. Thus, by selecting and determining a plurality of markers that are as far as possible from each other as important markers, the accuracy of content view control is increased.

(Process of step S06)
Next, in step S06, it is necessary to read the coordinate information on the guide sheet described in each important marker. Here, however, the marker on the guide sheet has a shape as shown in FIG. If the shape in the image is extremely deformed, it is difficult to read the coordinate information.

  Therefore, for example, as shown in FIG. 13, if the coordinate indicating marker is configured using a pattern by coloring, the reading process is relatively easy even if the shape is deformed.

  Referring to FIG. 13, the coordinate indicating marker 1200 is composed of an array 1201 to 1210 of small regions (cells) colored with a specific color and a black frame 1211, and serves as a reference for recognition. The cell 1201, the X coordinate information cell group 1202 to 1205 representing the X coordinate value of the center position of the reference cell, the Y coordinate information cell group 1206 to 1209 representing the Y coordinate value, and reading errors of those cell groups It includes a parity value display cell 1210 for detection.

  Here, the reference cell 1201 is colored red as described above, and the other cells are colored blue (bit value = 0) or green (bit value = 1) according to the bit value to be displayed. The parity value display cell 1210 is determined as an adjustment value for setting the total number of blue cells to an even number, so that if this rule is violated, it can be determined that the data is invalid due to a read error.

  As an example, if the cell color array for a certain marker is “red, (blue, green, blue, green,) (blue, green, green, green,) blue”, the center coordinates of the red cell are X = 0b0101 × 256, Y = 0b0111 × 256 (Here, the 4-digit number following 0b is a binary number). Further, since the total number of blue cells is 4, that is, an even number, it is determined as normal.

(Process of Step S07)
In step S07, a coefficient for converting the coordinates on the LCD 102 and the corresponding coordinates on the content 900 in one direction or each other is calculated.

  FIG. 14 shows a modeled state where an arbitrary point on the guide sheet 401 is projected as one point in the guide image on the imaging surface in the CCD camera 106. As will be described later, since accurate modeling inside the camera may not be necessary in this embodiment, this model takes a simple form like a pinhole camera.

  Referring to FIG. 14, the guide sheet surface Sg is mapped onto the imaging surface Sc inside the camera by plane projection conversion in a three-dimensional space.

Now when a point Pg on Sg is projected onto a point Pc on Sc, Pg = (x, y, z) under the appropriate coordinate system G, and the projection center point O (in FIG. Point) is O = (x0, y0, z0). Furthermore, under the coordinate system C in which the xy plane includes the imaging surface Sc, the center points O, Pc, and Pg with Pc = (u, v, 0) and O = (u0, v0, d) are on a straight line. Therefore, the following can be said.

  This means that the origin of the coordinate system C and the 3 × 3 rotation matrix R are appropriately determined, and the coordinate system G is translated in the z-axis direction so that a system in which the coordinate system G is rotated by R can be obtained. Then, it can be expressed using coordinate values as follows.

  The relationship that the point Pg = (x, y, z) on the coordinate system G is copied to the point Pc = (u, v, 0) on the coordinate system C by projection transformation is generally included even when z ≠ 0. Can be expressed by the above formula. Based on the above relationship, the problem of obtaining the parameters required to correlate the coordinate values of Pg and Pc from several sample points is a kind of camera calibration problem and is a solution called DLT (Direct Linear Transformation) method. It has been known. For such DLT method, for example, reference: Abdel-Aziz et al, Direct Linear Transformation from comparator coordinates into object space coordinates in close-range photogrammetry, Proc. Of the Symp. On Close-Range Photogrammetry (1971), pp .11-18, American Soc. Of Photogrammetry.

  In the modification of the present embodiment, the guide sheet surface Sg is the xy plane of the coordinate system G, and z = 0 is always set in Pg = (x, y, z). At this time, the DLT method results in solving an 8-ary linear equation based on the coordinate values on the guide sheet surface and the imaging surface for the four sample points, and is relatively fast in today's computers including portable devices. It can be solved.

  That is, when Pg = (xi, yi, 0) and Pc = (ui, vi, 0) for the i-th sample point, Expression Y is expanded to obtain the following expression for calculating Pc from Pg.

  Here, Ri, K1, K2,..., K8 have the following relationship.

  Solving Equation Z for K = (K1, K2,..., K8) T yields the following relational expression.

  Parameters K1, K2,..., K8 can be obtained by solving an 8-ary simultaneous linear equation obtained by juxtaposing equations W for four sample points. Then, Pc can be calculated from Pg by the formula Z. Although abbreviated here, the formula which transforms Formula Z and calculates Pg from Pc is also obtained.

  After all, if the parameters K1, K2,..., K8 are obtained, the coordinates on the imaging surface, that is, the guide image, and the coordinates on the guide sheet surface can be converted into each other by simple calculation. The coordinates and the coordinates on the content 900 corresponding to the guide sheet surface can be mutually converted by the same calculation. For example, in the present embodiment, similarly to the first embodiment, if the content 900 is bitmap data, each coordinate on the content corresponding to each coordinate on the LCD is obtained and the luminance data is stored in a memory corresponding to the LCD. If the content 900 is vector data, the coordinates of feature points on the content are converted into coordinates on the LCD, and then rendering is performed on a memory corresponding to the LCD.

  An actual camera may include a plurality of lenses, and when importance is attached to the accuracy of coordinate conversion, a more complicated method considering correction of optical distortion or the like can be used. However, in the modification of this embodiment, the change in the position of the guide sheet surface or the imaging surface by the user should be immediately reflected in the browsing content on the LCD. For example, if the change is continuously reflected at a predetermined time interval, the method of the present embodiment that can calculate the coordinate transformation at high speed is preferable. In addition, although a certain degree of accuracy is required for coordinate conversion, the coordinate position changes due to camera shake in addition to the intentional change by the user. Therefore, the coordinates obtained by the conversion are not reflected on the LCD as they are, which will be described later. It may be preferable and easy for the user to perform the camera shake correction.

  Note that the above description of the coordinate conversion method indicates that a known method can be applied to convert the coordinates on the guide sheet surface to the coordinates on the imaging surface. A conversion method may be used. For example, after the coordinate transformation parameter is first obtained using the DLT method, a change in coordinate value may be obtained using a method for obtaining a motion vector, which will be described later.

[Modification 2 of Embodiment 1]
The above embodiment is based on the premise that the guide sheet surface is a part of a virtual plane, that is, not curved. However, when the guide sheet is like a thin paper, it is more natural if the corresponding content appears to be folded according to the curve of the guide sheet. In such a case, the guide image is divided into rectangular small regions each having the coordinate indicating marker as a vertex, and the above parameters K1, K2,... The corresponding part of the corresponding part content may be mapped and displayed. As a result, the content appears to be folded along with the curve of the guide sheet.

  As the above application, a spherical surface can be used as a guide sheet. For example, using the content expression control device according to the present invention as a guide sheet in which coordinate information based on longitude and latitude is written on a spherical surface, the world map data represented by the Mercator projection is converted into a small square as described above. If mapping is performed for each area, it is possible to browse as if looking at the globe. In an actual globe, you can browse only one type of fixed world map. However, according to the present invention, by preparing multiple world map data by time axis or application, one sphere can be regarded as a display medium. It is possible to browse the globes for different purposes and browse weather maps that change over time.

  Similarly, it is also possible to map and use map data according to various uses by using a three-dimensionally shaped mountain model in which coordinate information based on longitude and latitude is written as a guide sheet.

[Modification 3 of Embodiment 1] (Variation of guide sheet)
Hereinafter, as a third modification of the first embodiment, other various guide sheet forms will be described with reference to FIGS. 15 and 16.

  15A shows the coordinates of each point on the guide sheet (indicated by “*” in the figure) by characters (including numbers) instead of graphics or color patterns. Reading expressed coordinate values as data can be performed by applying a character reading technique widely used in an apparatus called OCR (Optical Character Reader).

  FIG. 15B uses a free image as a guide sheet. That is, the same or similar data as the entire external reference image [FIG. 15 (b)] is held in advance in the content expression control device as a reference image for collation, and a part of FIG. By matching the pattern with a digitized guide image, it is possible to calculate which part is cut out as a guide image, that is, a view, and view control of the content to be displayed based on the calculation result. It becomes possible.

  As for the image used at this time, a pattern is written to every corner as much as possible, and the smaller the number of similar patterns for each part, the smaller the erroneous recognition, which is preferable.

  Note that any method may be used for pattern matching between the reference image for matching and the guide image. As a first method, each pixel constituting the guide image while changing the enlargement ratio in order from the upper left coordinates of the matching image data. There is a method of calculating a view by searching for a coordinate and an enlargement ratio at which a value obtained by taking the difference in brightness and dividing the difference by the number of pixels in the comparison portion is minimum. Also, as a second method, the image data for collation is divided into appropriate blocks, and the data obtained by performing discrete cosine transform on the luminance column for each color component of the vertical and horizontal pixels for each block and the discrete cosine transform in the same manner for the guide image. There is a method of collating the obtained data with frequency components.

  The second method is to first collate the entire guide image from a low frequency component, that is, a pattern of rough luminance change, and sequentially collate high frequency components hierarchically only when the difference is within a certain threshold, It is intended to eliminate unnecessary detail collation as much as possible and improve the efficiency of collation processing.

  Here, when a free image is used as a guide sheet as shown in FIG. 15 (b), it is possible to use a still image taken at the place of use, and there is an advantage that it is not necessary to carry a dedicated guide sheet. . For example, when browsing content on a business trip destination, a magazine cover or a part of newspaper may be photographed and registered as a collation image on the site and used as a guide sheet. In this case, the virtual size of the content can be determined based on the size of the photographed object. For example, if you fold a newspaper into A4 size, shoot and register it, and use it as a guide sheet, the content is mapped to the A4 size, so that the operator has the virtual size of the content. Perceived to be A4 version.

  Here, a relatively large guide sheet may be used for highly accurate view control, while a relatively small guide sheet may be used for performing a wide range of view control with a small number of operations. In this way, it is possible to arbitrarily select the accuracy of view control according to the purpose. Similarly, in the first embodiment, a plurality of types of scale guide sheets can be prepared and used according to the required accuracy. A plurality of types of scale guide sheets can be created, for example, by simply enlarging or reducing the original guide sheet.

  In addition, if it is possible to detect a motion vector that represents the movement of the guide sheet relative to the imaging surface of the camera from the changes in the guide images that are taken sequentially, it is possible to control the expression of the content that is expressed based on the detected motion vector. Also good. There is a lot of research in the field of moving image processing to detect the apparent motion (optical flow) of the luminance pattern and calculate the motion vector from it. The basic method is literature: JKAggarwal et al , On the computation of motion from sequences of images-a review, Proc. IEEE, Vol. 76, No. 8, pp. 917-935, 1988. Here, the motion vector may be detected even if the object serving as the guide is not a planar object or a rigid body. FIG. 16C shows an example in which the operator's one hand (the hand without the device) is used as a guide. In this case, a view may be calculated by performing pattern matching using a hand contour pattern or a hand wrinkle pattern, and a motion vector may be detected therefrom, or more generally based on an optical flow. A motion vector may be obtained. Further, in order to improve the accuracy of collation and detection, a graphic pattern characteristic to the skin of the hand or a coordinate indicating marker as shown in FIG. 6 may be written or a glove printed with a coordinate indicating marker may be used. Good. In these cases, in order to adjust the condition of the background light, a light that irradiates the same object may be provided near the camera 106 that shoots the object, and the object may be irradiated during shooting.

  FIG. 16 (d) shows an example in which a CCD camera is directed to the operator's own face and a face image is used as a guide when the photographing direction can be changed in a mobile phone device. Also in this case, since the recognition accuracy varies depending on conditions, a method of writing a coordinate instruction marker on the face skin or a method of using a mask or glasses on which the coordinate instruction marker is printed can be considered. As described above, when the operator's own face is used for view control, the front and rear are reversed, and the processing mode must be changed. At this time, as the device is moved away from the face, it is usually preferable to increase the enlargement ratio to match the intuition.However, when increasing the enlargement ratio, it is often desirable to see details. It may be preferable to raise it.

  In addition, there is a method in which a digital watermark including coordinate information is inserted for each part of the reference image, coordinate information is obtained by decoding the photographed digital watermark, and parameters relating to the expression of the content are controlled. An example of a method of inserting a digital watermark for each part of an image is disclosed in Japanese Patent Laid-Open No. 2001-109756. The publication proposes embedding a keyword for searching an image for each part into an image using a special digital watermark technique. However, when applied to the invention according to the present application, In order to express the coordinate information for each part, a digital watermark technique similar to that of the publication can be used.

  Although not shown, the coordinate information can be expressed by a change in color on the guide sheet. For example, when the color of a specific coordinate of the guide sheet is displayed by the ratio (R%, G%, B%) of each of the three primary colors of light, red, green, and blue, the upper left color (100% , 0%, 0%), the upper right color is (0%, 100%, 0%), and the red and green components are gradually changed as gradation from the upper left to the upper right, and the lower left color is (100% , 0%, 100%), the lower right color is (0%, 100%, 100%) The color will be unique. A part of this guide sheet is photographed as a guide image, and the color component of any pixel (such as a corner of the colored part) in the colored part in the guide image is separated on the guide sheet corresponding to that pixel. It is possible to estimate by calculating back the coordinates of the vertical and horizontal axes. For example, in the above example, if the color component of a certain pixel is (50%, 50%, 0%), it can be estimated that the pixel is near the midpoint of the upper side of the guide sheet.

  In such a case, the value of the recognized color component varies depending on conditions such as the brightness and color of the external illumination. For example, before the view control operation, the entire guide sheet is photographed and colored portions are It is preferable to store the color components of the four corners of each corner and use them as reference values for calculating the coordinates during view control.

[Modification 4 of Embodiment 1] (camera shake correction, etc.)
In the content expression control system according to the present invention, as described above, it is conceivable that the content display becomes difficult to see due to a slight movement of the positional relationship between the apparatus and the guide sheet, in addition to the intentional change by the user.

  Therefore, it is preferable to have a so-called camera shake correction function when shooting the guide sheet. Any method may be used for camera shake correction. For example, if the parameters for each view are calculated based on the guide images at multiple points in time and the view of the content is controlled using the value obtained by taking the average (moving average) of them, fine blurring on the display can be achieved. Is alleviated.

  In addition, it may be easier to view the guide sheet by partially restricting how the view moves. For example, as described above, rotation around any coordinate axis in the three-dimensional space between the CCD camera and the guide sheet increases the degree of freedom of browsing, but on the other hand, content such as characters may be difficult to see. Therefore, when considering a three-dimensional xyz coordinate system with the camera as a reference, rotation about any axis of x, y, or z may be limited. For example, it is assumed that rotation about the x and y axes does not affect the view control in principle, and rotation about −45 degrees to +45 degrees does not affect the view control in rotation about the z axis. Then, while the display is stabilized, the vertical and horizontal orientations of the view can be controlled. The display can also be stabilized by limiting the minimum time interval at which the content changes. Furthermore, if zoom control is temporarily or permanently stopped, the degree of freedom of view control is limited, but the stability of display is improved. As described above, display stability is particularly required for characters, so only the upper left coordinate of each character string is subject to control when the view is rotated, enlarged, or reduced, and the shape and orientation of each character are controlled. May be controlled so as not to affect. In addition, if the view movement in all directions and the rotation axis can be temporarily stopped by a key operation or the like, the display can be stabilized and easy to see. That is, referring to the flowchart of FIG. 7, in step S09, in addition to the determination of the presence / absence of an end instruction, the presence / absence of a pause instruction is determined. If there is a pause instruction, the return to step S01 is suspended until the instruction is canceled. Good.

[Modification 5 of Embodiment 1] (Camera Position)
In the above embodiment and its modification according to the present invention, it is preferable that the CCD camera 106 is positioned on the back surface at the center of the LCD 102 with reference to FIG. As a result, even when the device and the guide sheet are sufficiently close to each other, it is possible to perform view control as if looking through the magnifying glass with an intuitive movement operation.

  However, in an actual mobile phone device, the CCD camera may be located on the back surface of the peripheral portion of the LCD due to structural limitations.

  For example, referring to FIG. 17, consider a case where the CCD camera 1406 is located on the back surface 1406 of the lower right end 1407 of the LCD 1402. In this case, when the operator moves the center of the LCD to the vicinity of the upper left vertex of the guide sheet in order to see the vicinity of the upper left vertex of the content, the CCD camera takes a picture centering on the lower right corner from the upper left vertex of the guide sheet. become. As a result, the method according to the first embodiment brings about a feeling that the content is mapped in the upper left direction of the guide sheet. Therefore, in the case of conforming to the first embodiment, the calculation formula of the content coordinates (Xr, Yr) may be corrected by adding constant values α and β as follows.

Xr = Xw · (Xq−Xp) / Xv + Xp + α
Yr = Yw · (Yq−Yp) / Yv + Yp + β
In this case, however, if the user tries to look at the lower end or the right end of the guide sheet, the CCD camera will take an image outside the effective portion of the guide sheet, so care must be taken. In this case, referring to FIG. 18, there is a method of printing by extending or translating the coordinate indicating marker group in the lower right direction of the guide sheet. When the coordinate indicating marker group is simply translated in the lower right direction without changing the value of each coordinate indicating marker, the correction by α and β in the above calculation formula is not necessary.

  Further, referring to FIG. 19, a method of making one coordinate instruction marker correspond to a plurality of terminal models is shown. That is, a coordinate indicating marker group 1601 is arranged at the center of a large guide sheet 1600, and a shooting range determined according to the model of the operator's mobile phone device is indicated by a broken line 1602 or 1603. For example, if the CCD camera is a model located on the lower right back surface of the LCD, the operator uses the shooting range indicated by the broken line 1602, and if the model is located on the upper left back surface of the LCD, the operator uses the broken line. It may be determined such that the shooting range indicated by 1603 is used.

[Modification 6 of Embodiment 1] (Camera Position)
Alternatively, as shown in FIG. 20, the center of the camera LCD and the center of the camera may be matched using a mirror surface or a prism.

  FIG. 20A is a side view of the present embodiment, and FIG. 20B is a rear view. When the center positions of the LCD 1701 and the CCD camera 1702 in the mobile phone 1700 are different as shown in the figure, it is possible to correct the center line in the imaging of the CCD camera using the correction tool 1710. By fixing the mirror surface 1711 and the mirror surface 1712 inside the correction device 1710, the center line in the imaging of the CCD camera 1702 follows the path 1712 due to reflection by each mirror surface, and the center 1714 of the photographing range is the center position of the LCD 1701. Matches.

  Note that a known method can be used to fix the correction device 1710 to the mobile phone 1700, and thus description thereof is omitted here.

[Modification 7 of Embodiment 1] (Control other than view)
Here, it is shown that the control for content expression according to the present invention can be applied to the control other than the view control shown in the above embodiments and modifications.

In the first embodiment, the distance from the CCD camera 106 to the guide sheet A101 can be obtained by obtaining Xp and Xq (or Yp and Yq). That is, if the distance is d, d is proportional to Xq-Xp. For example, when the distance when Xq−Xp = Xv (same size display) is v,
d = v · (Xq−Xp) / Xv
Can be calculated as

  For example, in content such as a presentation composed of a plurality of pages, a value on the z-axis (that is, the distance between the device and the guide sheet = d) is used to determine the page to be browsed instead of determining the magnification rate of the content. be able to. Specifically, assuming that Pi (i = 1..n) is an array of pointers pointing to the head of data of each page of content, the subscript i can be calculated as follows.

i = n− [a · d]
(However, [] is rounded down, a is a constant, 1 ≦ i ≦ n)
At this time, the content enlargement rate is constant, but the distance from the CCD camera 106 to the guide sheet 401 (hereinafter referred to as a value in the z-axis direction with the origin on the guide sheet as a reference) is set as the content enlargement rate. It is only necessary to allow the operator to select one of the modes to be used for determining the page or the page to be browsed by key operation or the like. The content expression control device has a function to access a WWW (World Wide Web) page, the target indicated by the pointer is a URL (Uniformed Resource Locator), and the value in the Z-axis direction is based on, for example, browsing history It may be used for display control of a WWW page indicated by URLs ordered in advance.

  Similarly, for moving image content having a time axis, the z axis can be used to determine a display target frame or a reproduction start frame in the content. Specifically, this can be realized by replacing each page of the content with each display frame or each display start frame.

  Regarding the control of the page or the time axis in the content as described above, not only the z axis but also coordinate information in the x axis or y axis direction read by photographing the guide sheet may be used. For example, regarding moving image content, coordinate information in the x-axis direction may be used for fine control in units of display frames, coordinate information in the y-axis direction may be used for rough display scene control, and the z-axis direction may be used for display magnification. It may be used for control. In this case, since the center position of the display portion is constant, coordinate information in the x-axis or y-axis direction cannot be used for scroll control, but the x-axis or y-axis is used for content time axis control. It is only necessary to allow the operator to select one of the modes to be used for controlling the scroll or for controlling the scroll by a key operation or the like. Since the parameters assigned to the x-axis or the y-axis can be easily specified with absolute coordinates on the guide sheet, the parameters can be controlled more precisely than the parameters assigned to the z-axis.

[Modification 8 of Embodiment 1]
As described above, by acquiring a control value corresponding to the x-axis, y-axis, or z-axis by the control device according to the present invention, it is possible to control various parameters related to the expression of content using the control value. .

  For example, if the content according to the present invention is voice, sound, or music, parameters to be controlled for expression include playback position on the time axis, volume, sound image localization, playback pitch, or a specific acoustic effect. Degree etc. are mentioned. For example, for a specific sound source, it is possible to intuitively control the localization of the sound image on the plane by the position on the x-axis and the y-axis, and to control the volume by the position on the z-axis. Since the output of the control device according to the present invention is obtained as numerical data, a method well known to those skilled in the art can be used as a method for controlling the above-described parameters relating to sound using the numerical data. The detailed explanation is omitted.

[Variation 9 of Embodiment 1]
If the content controlled by the content expression control apparatus according to the present embodiment is a set of moving image data in which television broadcasts are collectively recorded on a hard disk or the like, for example, the x-axis is a broadcasting station or broadcast date, and the y-axis is The broadcast time zone and the z axis can control the time axis in a specific program, respectively.

  In this case, the guide sheet may be used also as a program list. That is, when a normal program list in which coordinate instruction markers are arranged vertically and horizontally is used as a guide sheet, a program search is facilitated. In this case, if the program list and the coordinate indication marker are printed in different colors, the image processing can be easily separated and the calculation cost can be reduced. For the same purpose, the coordinate indicating marker may be printed in a specific color on a transparent film and used by superimposing it on a program list supplied by a newspaper or the like. Again, if a control value corresponding to the x-axis, y-axis, or z-axis can be acquired by the control device according to the present invention, the broadcast date, broadcast time zone, or specific program of the video data to be reproduced on the hard disk Since it is obvious to those skilled in the art to control parameters such as the time axis, detailed description thereof is omitted here.

[Modification 10 of Embodiment 1] (Robot Camera)
The content expression control apparatus according to the present embodiment controls a view of a monitoring camera that can be remotely controlled. Here, the video imaged and output by the surveillance camera is considered as the content to be controlled. For example, by using a mechanism such as a stepping motor, there are monitoring cameras that can remotely control the vertical shooting angle, the horizontal shooting angle, or the zoom, but using the output of the content expression control device according to the present invention, The x-axis control value is used for the vertical shooting angle, the y-axis control value is used for the horizontal shooting angle, and the z-axis control value is used for zooming. It is possible to intuitively control the view of the surveillance camera by transmitting to the content expression control device at hand and displaying it on the LCD.

  FIG. 21 is a diagram illustrating a functional configuration of the content expression control system according to the modification of the first embodiment.

  Referring to FIG. 21, this system includes a content expression control device 1800, a remotely-controllable surveillance camera device 1810, and a guide sheet from which guide image data 311 is based. In the content expression control apparatus 1800, reference numerals 301 to 303 correspond to the units shown in FIG. From the expression control data generation unit 303, control values based on the above-described x-axis, y-axis, and z-axis are output, sent to the monitoring camera device 1810, and used to control each unit 1810 to 1813. The monitoring image input by the monitoring image input unit 1814 is sent to the display unit 1805 of the content expression control device 1800 as it is and displayed.

  When such a surveillance camera is mounted on a vehicle body or robot that can be moved by remote control, the position, movement direction, movement speed, etc. of the vehicle body or robot are determined using the output of the content expression control device according to the present invention. You may control.

[Modification 11 of Embodiment 1]
The content expression control apparatus according to the present embodiment can be used to move editing points when editing content. When a document editing device, a graphic editing device, a moving image editing device, or the like is collectively referred to as a content editing device, it can be said that the content editing device includes a function of a content browsing device.

  The editing point of the content editing apparatus is often handled as a cursor, but the output of the control apparatus according to the present invention can be used for such movement of the cursor. For example, an x-axis control value can be used for the vertical movement of the cursor, and a y-axis control value can be used for the horizontal movement. In this case, the content view may be moved according to the cursor movement, or may not be moved. In this case, the z-axis control value may be used for zoom control as in the previous embodiment, or may be used for content page or time axis movement control. When used for page or time axis control, the cursor indicating the edit point may be moved along the time axis.

[Modification 12 of Embodiment 1]
Furthermore, the above-described apparatus, that is, a content expression control method in a computer can be provided as a program. Such a program can be recorded on a computer-readable recording medium such as a flexible disk, a CD-ROM (Compact Disc-ROM), a ROM, a RAM, and a memory card, and provided as a program product. Alternatively, the program can be provided by being recorded on a recording medium such as a hard disk built in the computer. The program can also be provided by downloading via a communication network.

  The provided program product is read into a program storage area 231 such as a hard disk and executed. The program product includes the program itself and a recording medium on which the program is recorded.

  As described in the above embodiments and modifications, in the present invention, the zoom operation can be performed simultaneously or continuously with the scroll operation as a kind of view movement operation.

  Furthermore, according to the present invention, the operation can be performed while maintaining the portability of the entire system.

  In the present invention, it is possible to approach a specific object and precisely control its view. For example, as in Non-Patent Document 3 described above, at least one two-dimensional matrix code is sufficiently large within the photographing range of the camera in order to obtain sufficient accuracy in alignment by recognition of the shape of the code frame. The smaller the captured code image, the larger the quantization error in the recognition of the code frame shape, and the lower the accuracy.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  DESCRIPTION OF SYMBOLS 101 Mobile phone, 102 LCD, 106 CCD camera, 311 Guide image data, 301 Image data acquisition part, 302 Correspondence relation information acquisition part, 303 Expression control data generation part, 304 Expression data control part, 401 Guide sheet, 402 Coordinate indication marker , 700 guide images, 900 content, 901 views, 1710 corrector.

Claims (7)

A photographing means capable of photographing the subject by freely moving the device main body held by the user ;
An image display means for displaying content, which allows the user to freely move the device body held by the user and view the display surface ;
Reference image data storage means for storing reference image data for collation, in which the photographing means has photographed a reference object serving as a reference for display control of the content;
A guide image data input means for inputting data of a guide image obtained by photographing the part or the whole of the reference object by the photographing means;
The features of the input guide image and the reference image are collated, and the display range of the guide image relative to the reference image, which corresponds to which partial range of the reference image the guide image corresponds to, is specified. Image collation means for obtaining corresponding range information;
Display control data generating means for generating display control data for controlling at least a display range of a display image displayed by the image display means based on corresponding range information acquired by the image collating means;
Display control data output means for outputting display control data generated by the display control data generation means;
Display control means for controlling the image display means based on display control data output by the display control data output means,
Corresponding range information acquired by the image collating unit includes position information in the reference image of the guide image to be collated,
The display control data generated by the display control data generation unit includes position information that specifies a display range of the display image.   The content display control apparatus according to claim 1, wherein the display control data generation unit or the display control unit partially restricts a change in a parameter related to content expression with respect to a change in the guide image. The display control data generating means
Distance correspondence information calculating means for calculating distance correspondence information corresponding to the distance from the photographing means to the reference object based on the acquired correspondence range information;
Generating the display control data by using the distance associating information, the content presentation control apparatus according to claim 1 or 2. The content expression control device according to claim 3 , further comprising correspondence relationship switching means for switching the correspondence relationship between the specific display control data and the distance correspondence information. A shooting step that allows the user to freely move the device body held by the user and photograph the subject ,
An image display step for displaying content, which allows the user to freely move the device body held in his hand and view the display surface ;
A reference image data storage step of storing reference image data for collation, in which the shooting step images a reference object that is a reference for display control of the content;
A guide image data input step in which the shooting step inputs data of a guide image obtained by shooting a part or the whole of the reference object;
The features of the input guide image and the reference image are collated, and the display range of the guide image relative to the reference image, which corresponds to which partial range of the reference image the guide image corresponds to, is specified. An image matching step for obtaining corresponding range information;
A display control data generating step for generating display control data for controlling at least a display range of a display image displayed by the image display step based on corresponding range information acquired by the image collating step;
A display control data output step for outputting the display control data generated by the display control data generation step;
Causing the computer to execute a display control step for controlling the image display step based on display control data output by the display control data output step;
The corresponding range information acquired by the image matching step includes position information in the reference image of the guide image to be verified,
The display control data generated by the display control data generation step is a content expression control program including position information that specifies a display range of the display image. For content presentation control unit which controls the representation of the content, a reference object as a reference for presentation control of the content,
The site of the photographable surface from external, comprising a coordinate indicator marker indicating the coordinates on the plurality of coordinate systems,
The coordinate indicating marker is
A reference pattern for causing the content expression control device to recognize the coordinate indicating marker;
A coordinate information pattern representing coordinate information of the position of the reference pattern,
At least one of the reference pattern and the coordinate information pattern is colored with a specific color ,
The content expression control device includes:
Photographing means;
Image display means for displaying content;
A guide image data input means for inputting data of a guide image obtained by photographing the part or the whole of the reference object by the photographing means;
In the guide image input by the guide image data input means, important marker determining means for determining the position of the guide image corresponding to the reference object and determining at least two of the coordinate indicating markers as important markers;
Correspondence information acquisition means for acquiring correspondence information specifying a position corresponding to the guide image with respect to the reference object from the important marker determined by the important marker determination means;
Display control data generating means for generating display control data for controlling at least a display range of a display image displayed by the image display means based on the correspondence information acquired by the correspondence information acquiring means;
Display control data output means for outputting display control data generated by the display control data generation means;
The display control data output means based on the display control data outputs Ru and display control means for controlling said image display means, the content presentation control reference object. A content presentation control unit which controls the representation of the content, a content presentation control system including a reference object as a reference for presentation control of the content for the content presentation control unit,
The reference object is the site of photographable surface from external, comprising a coordinate indicator marker indicating the coordinates on the plurality of coordinate systems,
The content expression control device includes:
Photographing means;
Image display means for displaying content;
A guide image data input means for inputting data of a guide image obtained by photographing the part or the whole of the reference object by the photographing means;
In the guide image input by the guide image data input means, important marker determining means for determining the position of the guide image corresponding to the reference object and determining at least two of the coordinate indicating markers as important markers;
Correspondence information acquisition means for acquiring correspondence information specifying a position corresponding to the guide image with respect to the reference object from the important marker determined by the important marker determination means;
Display control data generating means for generating display control data for controlling at least a display range of a display image displayed by the image display means based on the correspondence information acquired by the correspondence information acquiring means;
Display control data output means for outputting display control data generated by the display control data generation means;
The display control data output means based on the display control data outputs Ru and display control means for controlling said image display means, the content presentation control system.

Images