KR100963679B1 - Display method of a virtual screen by movement of hand helded information equipment - Google Patents

Abstract

In the present invention, in order to overcome the limitations of the display size of the portable information terminal and provide an enlarged virtual screen, the conventional concept of moving the virtual screen by various input means such as a mouse or a keyboard on a fixed display is reversed. This is a method of moving or enlarging / reducing the virtual screen area required for the display window by recognizing the motion information of the information terminal when the display device of the portable information terminal is spatially moved. The method of the present invention allows the user to read the virtual screen more naturally since the display window can be moved as much as necessary to read or enlarge / reduce other areas of the virtual screen as if the newspaper is read with a magnifying glass. In addition, in order to recognize movement information of the display device, image information of a 3D gyro sensor or a camera mounted together may be used.

Virtual screen, information terminal movement, mouse movement, enlargement, reduction

Description

DISPLAY METHOD OF A VIRTUAL SCREEN BY MOVEMENT OF HAND HELDED INFORMATION EQUIPMENT}

The present invention relates to a display technology of a portable information terminal, and more particularly, by moving a display element of a portable information terminal to move / enlarge / reduce a virtual screen and move a virtual screen by moving a display element that moves a mouse cursor. It relates to a method of zooming in and out.

In recent years, as portable information terminals have become widespread and wired / wireless communication means have been rapidly developed, the multimedia service market providing various multimedia information using a portable information terminal has been rapidly expanding.

Despite the expansion of such multimedia services, portable information terminals, which are widely used at present, use small display elements in order to maintain portability. In other words, as the size of the display device increases, the readability of the information improves, but since the ease of portability decreases due to the increase in weight and size, the display device of the small size is inevitably used.

However, when such a small display device is used, the size of the unit pixel is limited and the resolution of the visible screen is greatly reduced, which is a major factor in reducing the readability of the user. For example, a mobile phone has a difficulty in converting a resolution of a photo or other information on a web into a separate WAP standard in order to provide web information in a very poor display environment.

In order to overcome this problem, a physical approach to minimizing the size of a unit pixel is attempted, but current technology faces a physical limitation to overcome this problem, and in reality, a high resolution, for example, a portable device of 1024 X 768 Even if the display device is developed at high cost, it is difficult to read small objects and letters with human eyes.

On the other hand, in order to overcome the limitations of the display screen, a technology for creating a virtual screen of which the resolution has been expanded in software and moving the virtual screen to the display window through the operation of a mouse or a keyboard has been developed and applied to the portable notebook in the past. However, such a conventional method has an advantage of showing an enlarged virtual screen on a small screen by manipulating a keyboard or a mouse, but there is an uncomfortable operation and an unnatural problem when moving between virtual screens.

Conventionally, in order to overcome the limitations of the small display screen, the technology of creating a virtual screen with the enlarged resolution in software and moving the virtual screen to the display window by operating a mouse or keyboard is intended to move the virtual screen while the monitor is fixed. It was a technical implementation based on the idea. However, the present portable information terminal, unlike the notebook or monitor of the past, since the information terminal itself is light and compact and can be easily moved, in the present invention, the concept of the past is reversely applied to be suitable for a portable information terminal.

The present invention has been proposed to solve the above problems, and stops the virtual target screen, such as when viewing a newspaper with a magnifying glass, and displays the contents of the required portion of the virtual screen by moving the information terminal itself attached to the display element An object of the present invention is to provide a method for displaying a virtual screen by moving an information terminal for placing a desired virtual screen on a mouse cursor fixed at a predetermined position of a display window and displaying information desired by a user by operating a click button.

In order to achieve the above object, the method of the present invention provides a display screen of a portable information terminal that is small in order to overcome the limitation of poor readability. CLAIMS What is claimed is: 1. A virtual screen providing method for moving a display window itself of the information terminal spatially so that a required area on a virtual screen can be viewed, the method comprising: setting a reference position of a display window of the information terminal; A second step of determining before and after movement information of the display window of the information terminal from the reference position to enlarge or reduce the virtual screen displayed on the display window of the information terminal; And a third step of determining up, down, left, right movement information of the display window of the information terminal from the reference position and moving the virtual screen displayed on the display window of the information terminal up, down, left, and right. It features.

 In the second step, when the information terminal moves forward from the reference position, the virtual screen displayed on the display window of the information terminal is enlarged, and when moved backward from the reference position, the information terminal is displayed on the display window of the information terminal. When the information terminal is moved to the left from the reference position, the third step is to move the virtual screen displayed on the display window of the information terminal to the left and to the right from the reference position. When the virtual screen displayed on the display window of the information terminal is moved to the right side, and moves upward from the reference position, the virtual screen displayed on the display window of the information terminal moves to the upper side, and when moved downward from the reference position. Virtualization displayed in the display window of the information terminal Move the face downward.

In addition, the second step or the third step, the space range in which the information terminal can be moved, and the resolution range of the virtual screen according to the previously set, and if the position of the information terminal is out of the set range, By stopping the change in the virtual screen, the user is informed that the display device is located at the edge of the virtual screen.

In addition, in order to achieve the above object, the apparatus of the present invention, in the portable information terminal to move the display window itself of the portable information terminal spatially without moving the virtual screen to view the required area on the virtual screen, the display, A memory storing content data to be performed; A display device for displaying contents stored in the memory according to a control signal; Position sensing means for sensing a position; Input means; When a setting key is input from the input means, the current position input from the position sensing means is set as a reference position of a display element, and if movement is detected, the current position detected from the position sensing means is compared with the reference position. Calculate the down, left, right movement distance and the front and rear movement distance, and when moving forward and backward, enlarge or reduce the virtual screen displayed through the display device according to the movement distance before and after, When the right movement, the control unit for controlling the display device to move the virtual screen displayed through the display device in accordance with the up, down, left, right moving distance up, down, left, right . The position detecting means is a gyro sensor or a camera.

In addition, the method of another embodiment to which the present invention is applied is a mouse cursor at a predetermined position of the display element of the portable information terminal to realize the mouse function by fixing the mouse cursor to a specific position on the display and moving the display element of the portable information terminal. Setting fixed to the position of; Moving the display device up, down, left, and right to move the virtual screen below the cursor fixed to the display when the cursor needs to be moved; Indicating that the cursor is positioned on the click bar when the cursor fixed on the display is positioned over the click bar on the virtual screen; And clicking the input means of the portable information terminal to click the cursor located on the click bar on the virtual screen.

The present invention allows the user to view the virtual screen unfolded in front of the user by moving the display window as much as necessary to view another area of the virtual screen, compared to the conventional movement of another area of the virtual screen to the display window by mouse or keyboard operation. It can make you feel as if you are moving.

In addition, the present invention first overcomes the limitations of the display window of the portable information terminal and expands it to a virtual screen, thereby allowing a menu or various information to be displayed in a wider space. Therefore, when the present invention is applied to a mobile phone, it is possible to provide information on a wide screen without increasing the size of the display device. Second, the present invention can continuously show the extended virtual screen in conjunction with the user's movement to move the information terminal up, down, left, right as if reading a newspaper with a magnifying glass, information when screen enlargement and reduction is necessary The screen can be enlarged and reduced by simply pushing or pulling the terminal to the front of the user, thus providing a level of naturalness very similar to how a person gets things or information in everyday life. Third, the present invention can be implemented by the user's natural movement without a separate key operation. Fourth, the present invention can identify the movement of the information terminal without additional hardware in the information terminal using the camera can be implemented at low cost. Fifth, the present invention can be applied to a virtual mouse or a virtual keypad can replace the mouse or keypad of the information terminal, enabling the miniaturization / lightening of the equipment and can reduce the cost.

The technical problems achieved by the present invention and the practice of the present invention will be more clearly understood by the preferred embodiments of the present invention described below. The following examples are merely illustrative of the present invention and are not intended to limit the scope of the present invention.

1. Display method of virtual screen using mobile information of portable information terminal

FIG. 1 is a schematic diagram showing moving a portable information terminal according to the present invention to display another area of a virtual screen on a screen.

As shown in FIG. 1, when the user 40 moves the information terminal 10 up, down, left, or right from a reference position, the virtual screen shown in the display window 20 of the information terminal 10 is shown. If the area of 30 is moved up, down, left, and right, and moved forward to move away from the user 40, the virtual screen 30 is enlarged and displayed on the display window 20, and when the user pulls back close to the virtual The screen 30 is reduced so that it is displayed on the display window 20.

Referring to FIG. 1, reference numeral 10 is an information terminal having a location recognition function, reference numeral 20 is a display window of a portable information terminal, and reference numeral 30 is a virtual screen.

Until now, the entire screen has been used in the size of the display window. For example, a mobile phone uses a QVGA graphics display with a resolution of 240 x 320 pixels. However, in the present invention, in order to overcome this limitation, a wider virtualization screen 30 is stored in a memory (for example, 1024 X 768 pixels) by using a graphic driver and a memory, and a moving position of the information terminal 10 is stored. The virtual screen area corresponding to 240 x 320 pixels of the corresponding position is recognized and displayed on the window of the display 20.

The present invention provides an effect as if the virtual screen 30 is present in the space of the position where the display device 20 of the information terminal is moved. Therefore, the virtual screen 30 is deployed on the front of the user 40 without the physical investment for increasing the size of the display device 20 or increasing the resolution, and the user magnifies the display device 20 of the portable information terminal 10. By moving like a window, a virtual screen with a resolution larger than that of an actual display device can be provided to a user. At this time, the size of the enlarged virtual screen 30 is set to be proportional to the distance the display device 20 extends toward the front of the user 40.

2 is a schematic diagram illustrating a virtual screen reduction / enlargement concept according to a movement of a display device before and after a display device having higher resolution than that of an original display is stored in a memory according to the present invention.

2, reference numeral 1 denotes a resolution Rd of a display, 2 denotes a resolution Rv of a virtual screen, and 3 denotes a resolution Rc of content stored in a memory. 4 is an enlarged virtual screen area shown enlarged than the resolution of content stored in the memory, and 5 is a reduced virtual screen area shown reduced.

In the case where the size of the virtual screen 30 is smaller than the resolution of the content stored in the physical memory, that is, in the reduced virtual screen area 5 close to the user, the display is limited by reducing the actual resolution of the content according to the position of the display device 20. When displayed on the window 20 and using a virtual screen having a larger resolution than the actual content resolution (enlarged virtual screen area 4), the display device 20 is enlarged by expanding the resolution of the content according to the position of the display device 20. Can be used.

The resolution of the content stored in the memory is called Rc (3), the resolution of the display device is Rd (1), and the resolution of the virtual screen that the user 40 selects by moving the display device 20 back and forth is Rv (2). ), When the display element 20 is at the position d1 which is the first position, the entire screen is displayed by reducing the screen of Rc (3) at the resolution of Rd (1). At this time, the reduction ratio is Rc / Rd = kL (k is a proportionality constant for the user's preference, L is the distance between d2, which is a position corresponding to the resolution of the content stored in the memory, and d1, which is the position of the first display element). . For example, if k is 2 and the resolution of the content is 10 times the resolution of the display element 20, the reduction ratio is 10 and the distance L is 5 cm. If d ≦ L, the resolution of the virtual screen is Rv = Rc / k (L-d), and if d> L, Rv = Rc (d-L) * k. At this time, the area of the virtual screen that can be displayed on the display device 20 shows only Rd / Rv, so that the remaining area can be seen by moving the display device 20 up, down, left, and right. If the resolution Rc (3) of the content is equal to the resolution Rd (1) of the display element, the position of the display element is the same as d2. Therefore, moving the display element to the front of the user is only possible in the area of digital enlargement 4. It works.

In fact, in the case where the method of the present invention is applied to an information terminal, it is not meaningful to extend the virtual screen indefinitely. In addition, when moving the display device 20 up, down, left, or right, the area outside the edge of the virtual screen is meaningless, and when the display device 20 reaches the edge of the virtual screen, the screen stops moving and no more screens are displayed. It will be necessary to inform the user that there is no. That is, it is necessary to stop the movement of the screen in an area outside of the areas A, B, C, and D set in advance.

3 is a flowchart of a screen magnification calculation and a screen display procedure according to a change in position of a display device after initial setting according to the present invention.

Referring to FIG. 3, the user sets the reference position, the operating distance L, and the maximum magnification ratio at the first use (S1). In order to reduce the computational load in the usual use of the information terminal, it is possible to select the start of the method according to the invention with a button switch or a keyboard.

When the display function according to the present invention is selected, a module for recognizing the position information of the display element is operated and notified to the processor, and the processor changes the ratio of the screen and the up, down, left, Calculate the position of the display device on the virtual screen according to the right change (S2, S3).

Using this information, the display controller enlarges or reduces a portion of the content located in the memory and sends it to the display device for display (S4, S5).

When the display device of the information terminal is pushed out in front of the user through the above method, a portion of the enlarged virtual screen is displayed on the display window of the portable information terminal, thereby providing an enlargement effect of the screen displayed on the window. Likewise, when the display window is pulled in front of the user, a part of the less enlarged virtual screen is shown through the window, so that the screen can be reduced in size. Therefore, in the present invention, the user makes a virtual screen proportional to the distance in front of the user like a newspaper, and moves and enlarges the screen in the same manner as viewing a newspaper with a magnifying glass. It can overcome the disadvantage of readability limitation.

2. Method for portable information terminal to recognize movement information of display device

end. Method using gyro sensor for 3D spatial position recognition

FIG. 4 is a block diagram of an information terminal providing a virtual screen using position information of a gyro sensor according to the present invention, and is an example of using a gyro sensor attached to a portable information terminal to provide a virtual screen.

Referring to FIG. 4, the information terminal according to the first exemplary embodiment of the present invention may include a gyro sensor 11, a processor 12, and a memory 13 storing display data for providing a position signal for detecting a current position. ), A display controller 14, an input unit 18, and a display 20.

The information obtained from the gyro sensor 11 is transmitted to the processor 12, and the processor 12 processes it to calculate the position information of the display element, and transmits the result of the calculation to the display controller 14 to the memory 13 A portion of the higher resolution screen data stored in is displayed on the display 20. The information terminal 10 of the first embodiment has the advantage of identifying the correct position, but should bear the increase in price, volume and weight due to the gyro sensor attached.

I. How to use the camera attached to the portable information terminal

FIG. 5 is a block diagram illustrating an information terminal providing a virtual screen after recognizing positional information using a camera according to the present invention. FIG. 5 is an example of identifying position movement information of a display device using a camera which is not normally used. .

As shown in FIG. 5, the information terminal according to the second embodiment of the present invention includes a camera 17, a controller 16, a dedicated signal processing device 15, a processor 12, a memory 13, and a display controller. 14, an input unit 18, and a display 20.

Referring to FIG. 5, the information input from the camera 17 is stored in the memory 13 via the dedicated signal processing device 15 for image processing. In the present invention, when the location information identification request is notified to the processor 12 by the input unit 18 such as a button of the information terminal, the processor 12 operates the dedicated signal processing device 15 or software module having the following functions. Position information of the display device is obtained by the following method.

1) Identification of up, down, left and right movement in 2D plane

An image input from the camera 17 is captured at predetermined time intervals (or frame units) to compare the screens, and thus the direction of movement is derived. One embodiment of the image processing method for this purpose is to select a reference image window 22 of NXM pixel size for comparison to the center region of the reference image as shown in Fig. 6 (e.g., 5 X 5 pixel size), this window While moving sequentially from the comparison image 23 to derive the difference between the comparison image and the reference image as much as the size of the window. The distance between the position having the smallest difference value and the center position of the reference image among the values thus obtained indicates the direction and distance 24 in which the display element is moved during the time interval. This method corresponds to the two-dimensional correlation technique in image processing technology. That is, assuming that the comparison image is Y (i, j) and the reference image is X (n, m), the resultant data W (i, j) of the two-dimensional correlation analysis is expressed by Equation 1 below.

W (i, j) = ∑∑ [Y (n + i, m + j) -X (n, m)], where [] is an absolute value

However, Σ is equal to the number of horizontal and vertical pixels N and M of the reference image pixel for n and m, respectively. Such movement information identification can be continuously applied to the front and rear frames of the acquired image to identify continuous position movement information.

2) Identification of movement in front of and behind the user of the display device

7 is a schematic diagram illustrating an example of setting a reference region and comparing the position movement information forward by comparing the reference region with the comparison image. A reference image window 22 of NXM pixel size is selected for comparison in the center region of the reference image, and the window is moved in order on the comparison image 23, and the difference between the comparison image and the reference image as much as the window size is determined. To derive.

When the camera 17 attached to the information terminal 10 moves back and forth, an image that is enlarged / reduced from the center of the screen is input to the input image. Therefore, when the comparison image is gradually reduced / enlarged and compared with the reference image, the image enlargement / reduction state of the state with the smallest difference indicates the forward and backward movement information of the display element.

If the size or zoom of the reference image window is precise, more accurate results can be obtained, but the adjustment of the window size and reduction / enlargement precision is applied within the range that the calculation time in hardware due to the increase of the calculation amount is within the time interval between images. Of course it should.

3. How to implement virtual mouse and keyboard with display device movement information

end. How to use virtual cursors

As portable information terminals have been reduced to the size of the mouse of the past, the size of peripheral devices has had a significant impact on the design of the equipment. Therefore, in order to overcome such a problem, the present invention can implement a virtual mouse and keyboard using the above-described virtual screen providing method.

First, the mouse cursor is fixedly overlaid on one place (eg, the center) of the information terminal 10 display device 50 (hereinafter, referred to as a virtual cursor). However, if necessary, the virtual cursor 50 can be prevented from covering the normal screen by programming the display element to appear only when the display device is spatially moved. Second, the information terminal 10 is moved so that a position requiring a click in the virtual screen 52 shown below is under the virtual cursor 53 (eg, the center) on the display. Third, the virtual cursor 53 is clicked by clicking the button located on the information terminal 10.

The method of the present invention is a reverse of the conventional concept of fixing the display and the screen, and moving the mouse. That is, the present invention is a method of moving the virtual screen by placing the mouse cursor in one place of the display device and moving the display device, and moving the place where a click is required under the virtual cursor. In addition, the method of the present invention can be applied to a virtual keyboard replacing a keyboard by applying the same concept to a keyboard or a keypad. For example, the keypad can be displayed on the screen of a mobile phone or PDA, and the mobile phone or PDA can be moved up, down, left, or right so that the required key is placed on the virtual cursor and clicked to replace the physical keypad.

I. How to configure extra virtual screen for virtual cursor

In order to actually apply the method of the present invention, when the virtual screen is moved at an absolute position on the display, it is difficult to move the virtual screen below the virtual cursor at the edge of the virtual screen. Therefore, in order to overcome this problem, an extra area is displayed on the virtual screen outside the actual screen so that the virtual cursor can be moved to the edge of the actual screen when the display device is moved. However, this method requires extra memory space for virtualization and can make the screen look unnatural.

In order to reduce the extra space of the virtual screen, the information terminal can be pushed out to the user to enlarge the virtual screen, thereby reducing the space separation effect between the display edge and the virtual cursor. For example, when applied to an actual product, in a virtual screen enlarged 4 times, if the extra space is 1/4 of the display width, the edge of the screen may be moved under the virtual cursor positioned at the center.

According to the method proposed in the present invention, an interface device such as a mouse or a keypad can be replaced by software, thereby realizing a reduction in manufacturing cost, miniaturization, and light weight.

All. Click to move the virtual cursor using the display movement information

When the virtual cursor is used in the original screen without enlarging the virtual screen, the display itself may be used as a mouse by moving the virtual cursor on the screen by linking the movement information of the display device to the virtual cursor. This method can be used to eliminate the disadvantages of giving extra space to move the edges described above under the virtual cursor. That is, if the display window shows the edge of the virtual space, the virtual screen stops moving further, and from this point on, the virtual cursor moves the virtual cursor instead of indicating the extra space of the virtual screen. Allows you to click on the click bar located at. Of course, the user's convenience can be supplemented by adjusting the display movement speed and the virtual cursor movement speed on the screen.

The present invention has been described above with reference to one embodiment shown in the drawings, but those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom.

1 is a schematic diagram showing moving the portable information terminal according to the present invention showing a different area of the virtual screen on the screen,

2 is a schematic diagram illustrating a virtual screen reduction / enlargement concept according to the forward and backward movement of a display device in a state where high resolution data is stored in a memory according to the present invention;

3 is a flowchart of a screen magnification calculation and a screen display procedure according to a change in position of a display device after initial setting according to the present invention;

4 is a block diagram illustrating an information terminal providing a virtual screen using position information of a gyro sensor according to the present invention;

5 is a block diagram of an information terminal providing a virtual screen after recognizing location information using a camera according to the present invention;

6 is a schematic diagram illustrating an example of identifying a position movement information by setting a reference region and comparing the same with a comparison image according to the present invention;

7 is a schematic diagram illustrating an example of setting a reference region and comparing the position movement information forward by comparing the reference region with the comparison image according to the present invention.

8 is a conceptual diagram illustrating the movement of the virtual cursor on the screen according to the display movement in a state in which the virtual cursor is fixedly fixed to the display device according to the present invention.

DESCRIPTION OF THE REFERENCE NUMERALS

10: information terminal 11: gyro sensor

12: processor 13: memory

14: display controller 15: dedicated signal processing device

16: controller 17: camera

20: display element 30: virtual screen

40: person

Claims (9)

In order to overcome the limitation of readability due to the small display window of the portable information terminal, the display window of the information terminal can be spatially moved without moving the virtual screen in displaying the extended resolution virtual screen on the display window. A method of displaying a virtual screen for viewing a desired area of an image, A first step of setting a reference position of a display window of the information terminal; Determining the position movement information in the forward and backward direction of the image captured by the camera of the information terminal, extracting the forward and rear direction movement information of the display window of the information terminal, and the information according to the extracted front and rear movement information A second step of enlarging or reducing the virtual screen displayed on the display window of the terminal; Determining the position movement information in the up, down, left and right directions of the image captured by the camera of the information terminal to extract the up, down, left and right direction movement information of the display window of the information terminal, A third step of moving the virtual screen displayed on the display window of the information terminal in the up, down, left and right directions according to the down, left and right movement information; The mouse cursor is fixed at a predetermined position of the display window of the information terminal, and a click button for operating the mouse cursor is provided at one side of the information terminal, and the click bar on the virtual screen of the display window is moved by the movement of the information terminal. Positioning a mouse cursor to display the virtual screen information of the click bar by operating the click button; and displaying the virtual screen by the movement of the information terminal. The method of claim 1, wherein the second or third step, The space range in which the information terminal can be moved and the resolution range of the virtual screen are set in advance, and when the position of the information terminal is out of the set range, the user stops the change of the virtual screen further so that the user can stop the virtual screen. And displaying the virtual screen by the movement of the information terminal. delete delete delete delete delete delete delete

Images