WO2007000491A1

WO2007000491A1 - Method, device, and program product for visually indicating a selected object, using a device equipped with camera

Info

Publication number: WO2007000491A1
Application number: PCT/FI2006/050279
Authority: WO
Inventors: Joonas Paalasmaa
Original assignee: Nokia Corporation
Priority date: 2005-06-29
Filing date: 2006-06-26
Publication date: 2007-01-04
Also published as: EP1896897B1; EP1896897A1; FI20055365A; FI20055365A0; EP1896897A4

Abstract

The invention relates to a method and device for visually indicating a selected object (P) in a desired subject (M) using a device (10) equipped with a camera (16) , in which an image is taken of the subject, and is analysed in order to find the occurrence of the selected object (P) from the bit image, which image is analysed in order to find the occurrence of the selected object (P) in the image and the orientation parame- ters of the indicator are calculated.

Description

METHOD, DEVICE, AND PROGRAM PRODUCT FOR VISUALLY INDICATING A SELECTED OBJECT, USING A DEVICE EQUIPPED WITH CAMERA

The present invention relates to a method, device, and program product for visually indicating a selected object in a desired subject, in which a bit image is taken of the subject, and is analysed in order to find the occurrence of the selected object form the bit image.

It is quite difficult to find a name and its related telephone number from a densely written list. Similarly, it is difficult to search for a specific street on a map. Another difficult situation is when someone has a map in their hand, but does not known where they are on the map.

The present invention is intended to create a new method and device for solving the aforementioned problems . The characteristic features of the method according to the invention are stated in Claim 1 and correspondingly the characteristic fea- tures of the device are stated in Claim 8. In addition, the characteristics of the program product designed to implement the method are stated in Claim 12.

The invention combines a camera and projector, in such a way that some indicator, for example, a frame, point, or arrow can be projected in the subject photographed by the camera. The subject can be, for example, a list or map. In the method, the camera is first of all used to take an image, which is analysed and, on the basis of the analysis, accentuation is re- fleeted by a projector onto the imaged subject. In this case, OCR (Optical Character Recognition) technology, which is as such known, can be exploited to detect text from a bit image. In OCR scanning according to the invention, the bit positions corresponding to the detected text are also recorded. Equally, it is possible to use other known pattern recognition, as long as the pattern can be defined and it can appear in the subject .

The procedure can operate as follows. Because the camera and projector are oriented in the same direction, i.e. the viewing images point in the same direction, the projection can be created in the same direction in which the detail is seen in the camera. The projected image does not appear in precisely the same location as the detected detail, because the projector cannot be in precisely the same place in the device as the camera is. It is possible to determine how many degrees the position of the image differs from that of the projection and arrange accordingly the angle of the indicator being projected. The distance of the subject from the device can be de- fined for this correction. If several distances are defined from different sides of the subject, a virtual three- dimensional model of it can be created.

OCR technology can be used to scan typewritten text in the im- age area into a character form. At the same time, a sufficient number of the bit positions of the detected text are stored.

After this procedure, it is possible to perform text retrieval from the scanned text using selected character strings, for example a name. When a hit is found, the bit position corre- sponding to it is recorded for later use.

The user can input text, speech, or other input to the search application. The input methods can be augmented by using scanned text data or images (pattern) . When a hit is found from the scanned text, a speech recognition element, for example, can be used to limit the number of alternatives in connection with ambiguous situations. Use of the search application ranges from searching for contact data in a dense list to finding a specific street on a map.

In the following, the invention is examined with reference to the accompanying drawings, which show some embodiments of the invention.

Figures Ia and Ib show the use of the device according to the invention in determining a location on a map,

Figures 2a and 2b show a way to search for a name from a list,

Figure 3 shows the device being used in levelling,

Figure 4 shows the mutual geometry of the camera and the projector, and

Figure 5 shows a flow chart of a search for a name from a list.

In the figures, the device according to the invention is marked with the reference number 10. It can be a mobile station or a PDA device. In the device 10, there is generally a microprocessor, a program and data memory, a display 14, and a keypad 12 and/or other I/O means. In terms of the present invention, it is essential that there is a camera 16 and a miniature laser projector 20 in the device. In the application of Figures Ia and Ib, there are also GPS positioning means in the device. In the device 10, there is an operating system, in which an application program is installed, which performs imaging and projection, as well as image analysis.

The miniature laser projector shown is made by Symbol Technologies Inc., USA (http://www.symbol.com/products/oem/lpd.htmlftl) . In the application of Figures Ia and Ib, the user has a map M, the position of the imaging of which is input to the application (not shown) . This can take place either by entering me- chanically the values of the detectable co-ordinates, or even by reading such information directly from the map, with the aid of OCR recognition. In Figure Ia, once the user has started the application, it takes an image of the map, which it stores as a bit image in the memory. On the other hand, the absolute position of the device 10 can be identified using a GPS-positioning device. The application calculates the position data to the bit image of the image, from which it is possible in turn to calculate the parameters for the optical indicator, in this case for the control of the laser projector. According to Figure Ib, the device shows the position "You are here' on the map M, with the aid of a laser dot, square, or other similar mark.

The above embodiment can be adapted, for instance in such a way that, instead of the GPS positioning, the local street name is entered and, on the other hand texts, i.e. character strings are detected from the bit image using OCR technology, and are stored with the position data. The input is compared with the OCR-recognized texts, in which case a hit will give the position data of the street in question. With the aid of this, the projection parameters are calculated as above.

In Figures 2a and 2b, a task is resolved, in which a telephone number is sought from a dense list of names L. According to Figure 2a, an image is taken of the list of names and stored as a bit image in the memory. The application can then perform effective OCR recognition and store and the names and telephone numbers as character strings, as well as the position in the bit image of each character string. The selected name is given as input, which the application retrieves from the stored character strings. The character string according to the hit also gives the position data, by means of which the control parameters are defined, for showing the laser image (for example, a rectangle) at the name.

In preferred examples, the precision of the projection can be increased by re-imaging the subject after analysis, which retrieves the change in position of the camera immediately prior to projecting. An approximate and thus rapid image analysis is sufficient for this.

A virtual ruler is shown in Figure 3.

When the shapes of the objects are then detected in the field of vision of the camera, it is possible to calculate how a ruler would be positioned on the planar surfaces of the shapes and a ruler can be projected onto the surface. If information on the orientation of the camera is added to this, the ruler

BP can be set on the surface either horizontally or verti- cally, as required. The orientation information is obtained either from the rest of the structure, or by using an angle sensor 22 contained in the device.

The fact that the projector and camera can never be placed at the same point, as can be seen from Figure 4, leads to a certain problem. In the device 10, the lenses of the camera 16 and the projector 20 are at a distance from each other. The camera 16 has an angle of vision 16' while the projector 20 has an uncorrected projection angle 20'. The corrected projec- tion angle 20" strikes the same area in the subject M only at a specific distance. Thus, it is useful to measure the distance, in order to increase the precision of the projection. According to the measured distance, the projection can be rotated so that the centre of projection and the camera's imag- ing centre coincide. With the aid of the distance measurement, the power of the laser projector can also be adjusted optimally. The projected dot or image can be exploited in the distance measurement. The distance of the subject can be measured with the aid of a simple geometric equation, if the projection angle and imaging angle (from the image positions) and the geometry of the device are known.

The operation of the device takes place with the aid of an application program arranged on top of the operating system. The flow chart of Figure 5 shows the operation of Figures 2a and 2b. Once the program has started it branches into an imaging function 100 and an input function 101, in which the user is requested for a name from list that is displayed. In the imaging function 100, the user is expected to aim the camera at the list and trigger imaging, in which case the image that is created is stored in the device's memory as a bit image. This is followed by text recognition, stage 102, in which OCR technology is used to search for written characters, i.e. lines of text (or other agreed patterns) , from the bit image stored in the memory area. The detected character strings and the bit positions corresponding to them are stored.

After this, the name input is compared with the text in the text lines, stage 103. This is a simple text-search routine. In connection with a hit, the position values corresponding to the hit are also stored. These values are used to define the orientation parameters of the laser projector, stage 104.

Finally, a command is given to the laser projector to project a preselected pattern onto the subject, stage 105.

In an improved variation, before projection the angle position of the subject is detected using approximate and thus rapid optical detection. This is used to correct the orientation pa- rameters, if the device has turned relative to the original imaging position.

The method according to the invention can be implemented using a program product in a device, which comprises a computer with memories and an I/O element. The program product comprises a computer-readable memory medium, in which there are computer- readable program components, which are characterized in that the said components comprise - a first component to be run programmatically for controlling the camera for taking an image and for storing the image obtained as a bit image of a selected size in a selected memory area of the said memory, a second component to be run programmatically for analys- ing the said bit image in order to find a selected object image from the bit image and for storing the bit positions of the found object image, and a third component to be run programmatically for calculating the orientation values form the said bit positions and for transmitting them to the said control unit, in order to orientate the optical indicator according to the found position of the object image.

In one embodiment, the said second program-code component con- tains a routine for entering text describing the object, and a routine for performing OCR text recognition in order to find text objects from the bit image and at the same time for storing the positions of the found text objects, and - a routine for comparing the text describing the object with the detected text objects and for picking the bit positions of the found text object for use in the said third component to be performed.

Claims

1. Method for visually indicating a selected object (P) in a desired subject (M) using a device (10) equipped with a camera (16) , in which an image is taken of the subject, and is analysed in order to find the occurrence of the selected object (P) from the bit image, characterized in that the device (10) is equipped with an orientation-controllable optical indicator (20), and the projection of the occurrence of the object (P) in the image and according to it the orientation of the indicator (B) directed to the object (P) are defined, on the basis of which definition the object (P) is indicated in the subject (M) using the optical indicator (B) .

2. Method according to Claim 1 for finding a position (P) on a map (M) , characterized in that the device (10) is equipped with positioning means (GPS) , in which case the geographical location of the map (M) is detected with the aid of analysis, according to which positioning data the projection of one's own position (P) and the corresponding orientation data of the optical indicator (B) are calculated.

3. Method according to Claim 1 for finding one's own position on a map, characterized in that information depicting one's own position is given as an object, which is optically detected from the subject formed by the map.

4. Method according to Claim 1 for finding a selected text ob- ject from a subject, characterized in that the analysis com- prises OCR text recognition, in which case the text objects of the subject are detected while at the same time defining their position data inside the subject and comparing each found text object with the selected text objects, when the orientation data of the optical indicator are defined according to the found text object.

5. Method according to Claim 1 for creating a virtual ruler in a subject formed by a selected surface, characterized in that a set of co-ordinates according to the device are selected and the optical plane (BP) at the selected angle is indicated according to it.

6. Method according to Claim 5 , characterized in that the device is equipped with an angle sensor (22) for determining an absolute set of co-ordinates.

7. Method according to any of Claims 1 - 6, characterized in that the optical indicator is a miniature-type laser projector, by means of which a dot, line, or other similar geometri- cal pattern is indicator in the subject.

8. Device for visually indicating a selected object in a desired subject, which device includes a camera and image analysis means for searching for the selected object from an image, characterized in that the device also includes an orientation- controllable optical indicator with control means, in which case on the basis of the definition data of the position of the occurrence of the object the control means are arranged to define the orientation direction of the indicator and accord- ingly indicate the object visually using the said indicator.

9. Device according to Claim 8 for finding one's own position on a map, characterized in that the device also comprises positioning means (GPS) and the analysis means are arranged to detected the geographical position of the map from the map image and the control means are arranged to define the corresponding data of the optical indicator on the map.

10. Device according to Claim 8 or 9 , characterized in that it includes OCR text-recognition means for recognizing text ob- jects from the subject while at the same time defining their position data and means for comparing each found text object with the selected text object and thus for defining the orientation data of the optical indicator.

11. Device according to Claim 8 for creating a virtual ruler in a subject formed by a selected surface, characterized in that the device comprises an angle sensor and control means are arranged to be controlled using the said angle sensor.

12. Device according to any of Claims 8 - 11, characterized in that the optical indicator is a miniature-type angle sensor.

13. Program product for implementing the method according to Claim 1 in a device comprising a computer with memories and an

I/O element, which program product comprises a computer- readable memory medium, in which computer-readable program components are stored, characterized in that the said components comprise - a first component to be run programmatically for controlling the camera for taking an image and for storing the image obtained as a bit image of a selected size in a selected memory area of the said memory, a second component to be run programmatically for analysing the said bit image in order to find a selected object image from the bit image and for storing the bit positions of the found object image, and a third component to be run programmatically for calculating the orientation values form the said bit po- sitions and for transmitting them to the said control unit, in order to orientate the optical indicator according to the found position of the object image.

14. Program product according to Claim 13 , characterized in that the said second program-code component contains a routine for entering text describing the object, and - a routine for performing OCR text recognition in order to find text objects from the bit image and at the same time for storing the positions of the found text objects, and a routine for comparing the text describing the ob- ject with the detected text objects and for picking the bit positions of the found text object for use in the said third component to be performed.