DE102016119071A1 - image capture - Google Patents

Abstract

The invention relates to a method for image acquisition, a system for image acquisition and a computer program product executable on at least one processor that runs on a data processing device and performs at least some steps of the method.

Description

The invention relates to a method for image acquisition, a system for image acquisition and a computer program product executable on at least one processor, which runs on a data processing device and performs at least some steps of the method.

There are already technologies in which images can be captured with image acquisition devices and then marked on a screen image areas and processed further. These previous technologies have the disadvantage that only those image areas can be marked, which can be found completely on the screen at a certain time. The marked areas may be obscured during marking by the finger or stylus.

There are also algorithms for the solution of the so-called "Visual Simultaneous Localization and Mapping (vSLAM)" problem, in which three-dimensional coordinates are detected with an image acquisition device. This technology uses the images of a moving camera to both estimate the current position of the camera against the visible 3D world and to reconstruct the visible 3D world from the current position of the image.

WO 97/48037 A1 describes a graphical user interface.

US 2010/0031041 A1 describes a system for capturing text, whereby only one section of text can always be detected, which can be displayed completely in a display at a certain point in time.

US 2011/0285810 A1 describes a method for capturing a picture panorama.

The object of the present invention is therefore to provide a technology with which one can easily and precisely select and process image sections from a camera image that are larger than half the display or even larger than the display.

• a. das Bilderfassungsgerät erfasst eine kontinuierliche Folge von Bildern, die auf der Anzeige dargestellt werden,
• b. durch Betätigung eines Bedienelements wird ein erster Punkt auf einem ersten Bild markiert,
• c. der Aufnahmebereich des Bilderfassungsgeräts wird verändert,
• d. durch Betätigung eines Bedienelements wird ein zweiter Punkt auf einem zweiten Bild markiert, und
• e. das Datenverarbeitungsgerät errechnet wenigstens aus dem ersten Punkt und dem zweiten Punkt einen Bildbereich.

The object underlying the invention is achieved in a first embodiment by a method for image acquisition for execution on a system for image acquisition, comprising at least one image capture device, a display and a data processing device, comprising the following steps:

• a. the image capture device captures a continuous sequence of images displayed on the display,
• b. by actuating a control element, a first point is marked on a first image,
• c. the recording area of the image acquisition device is changed,
• d. by pressing a control element, a second point is marked on a second image, and
• e. the data processing device calculates an image area at least from the first point and the second point.

With the method according to the invention, image areas can be selected very easily and optionally further processed. For example, these image areas may be larger than the display. For example, users of smartphones can record images that are much higher resolution than the resolution of the camera would allow and can also record images with a larger coverage than that of the camera. At the same time, the user is not limited to expanding images only in a (mostly horizontal) direction, as is known for example from the panoramic functions of smartphones. Another advantage of the invention is that the image area is not obscured during selection (such as with pins or fingers in the prior art). In addition, the geometry of the selection is freely definable with the method according to the invention - the height and width are freely selectable and you can also select freeforms. With the method according to the invention, for the first time in a single work step (for example touching and releasing a touch screen for selecting the image area), an image area can be defined in order to generate an image which then automatically also annotates (for example an arrow from top right to bottom left). and can be sent, for example, on social media.

Image capture device and data processing device

The image capture device may preferably be a camera. Alternatively, however, the image acquisition device may also be an endoscope (for example for medical applications). The camera may preferably be integrated in the data processing device (for example smartphone or glasses).

The data processing device preferably comprises at least one processor, main memory, at least one storage medium and interfaces for communicating with the display and the image acquisition device, for example.

The storage medium of the data processing device is typically a non-volatile memory, such as a flash memory or a hard disk. The flash memory may be a semiconductor memory integrated in the data processing device. It may alternatively or additionally be an external memory, which is realized by a pluggable into the data processing device digital memory card, such as by an SD memory card for "Secure Digital Memory Card".

Typically, such computing devices have a random access memory (RAM) in which at least portions of computer programs, operating system routines, applications, as well as images, videos, music data, and other data are loaded and executed or processed by the processor from the storage medium. The working memory can be non-volatile or volatile.

The method is preferably carried out with the aid of an application on the data processing device. An application - also referred to in short as an "app" - is a computer program, ie software that can be executed by the processor in the data processing device. It also has program steps in order to be able to access data from the processor on other apps, the operating system or on peripherals of the data processing device.

Central unit of the data processing device is preferably at least one processor, such as an A5 processor from the company ARM. It is preferably in terms of data technology and / or signaling with the image acquisition device (for example camera or endoscope), with an output unit (monitor) or combined input / output unit (for example touchscreen) and connected to the storage medium (for example flash memory).

The data processing device may be, for example, a computer, and preferably a desktop PC, a laptop, a pair of glasses or a smartphone.

Preferably, the display is integrated in the data processing device and / or the image acquisition device.

The image capture device may preferably be integrated in a device with the display and the data processing device. For example, it can preferably be a smartphone or a pair of glasses in which all three components are integrated in one device.

Alternatively, the components can preferably also be realized separately from one another. For example, in an endoscope, the image capture device may be separate from the display and the computing device.

operating element

The control in step b. and the operating element in step d. may be identical or different, for example. The operating element may preferably be a touchscreen (for example touch-sensitive display) or a button. The operating element may preferably also be only a part or a specific location of the touchscreen. This location may be marked on the display, for example. Preferably, the operating element in step b. and the operating element in step d. identical and more preferably a touch screen. An advantage of a touch screen is that the method according to the invention can be combined with gestures for controlling the data processing device.

The operation of the operating element in step b. For example, it may be preferable to activate (e.g., press the touch screen or button) and / or operate the control in step d. deactivating (e.g., releasing the touchscreen or button).

If the operating element is a touchscreen, it may preferably be identical to the display. The display can therefore preferably be the touch screen and thus at the same time the operating element.

Preferably, an operating element between the steps b. and d. at least once, most preferably at least ten times served to mark additional points as intermediate points. For example, the contours of an image area to be acquired can be traced in order to capture an image area with a complex geometry. Preferably, the control element after step b. also be held down, so that until you release in step d. continuously pixels can be marked as intermediate points (Lasso effect, Lasso tool).

Preferably, at least the first point and / or the detected intermediate points are displayed on the display (preferably in color). In this case, it is also possible to calculate shapes directly from these points and to display them on the display. These shapes can be, for example, rectangles, connecting lines between the points, Bezier curves and much more.

The positions of the first point, the second point and / or the intermediate points are preferably stored.

The positions of the first point, the second point and / or the intermediate points are preferably indicated on the displayed image on the display. The positions of dots are preferably defined in relation to the overall picture or preliminary overall picture.

Preferably, an overall image of individual images of the image sequence before step b. and / or after step b. calculated. Before step b. and / or after step b. can also be calculated a preliminary overall picture. Below in this application, the overall picture and the preliminary overall picture are explained in detail.

If image disturbances (for example reflections) are detected in the images, these image disturbances can preferably be removed from the resulting (provisional) overall image so that images which have acquired the same region of the overall image and do not have this image disturbance produce the (preliminary) overall image be involved.

The positions of the first point, the second point and / or the intermediate points are preferably transformed into the overall image and / or provisional overall image, and particularly preferably stored with the overall image and / or provisional overall image.

image capture

Continuous within the meaning of the invention preferably means that the images of the image sequence are recorded in temporal sequence relative to one another by the image acquisition device.

In step a. For example, only a section, an enlargement of an enlargement or an overall picture of the pictures captured in the picture sequence can be displayed on the display. In addition to this, further picture elements such as cursors, operating elements, pictograms, menus and other elements can be displayed on the display.

In step a. Preferably, at least one object can also already be recognized (see below). Depending on the type of object (for example letters), at least one further step (for example text recognition) may preferably be performed or offered to the user in a context-sensitive menu. If the at least one object is detected, then the result of this recognition (for example, words detected) can be displayed directly on the display (for example, underlining of the words). If the further step is carried out (for example text recognition), the result of this step can preferably be displayed on the display.

Preferably, in step b. and / or step d. the first and / or second point of the point in the first and / or second image, which is arranged under a cursor marked on the display at the time of operation of the respective operating element.

Preferably, in step b. Press the touchscreen until it is in step d. is released again. The cursor is preferably displayed on the display. Especially preferred is the cursor before step b. displayed on the display. This allows you to mark a point without covering the point with your finger or stylus. The (suitable) position of the cursor can preferably be calculated by the data processing device. The position or shape of the cursor can also be influenced by objects recognized in the image (see below section Object detection, example: cursor height adapts to a recognized word height or cursor jumps to the position of a detected nearest word).

Preferably, in steps a., B. and / or d. Real-time images displayed on the display. Preferably, objects are also recognized during real-time image display in real time (as described below). These objects are preferably identified in real time on the display. Preferably, a result of object processing (eg, text recognition) is also displayed on the display in real time on the display.

Preferably, during steps a. to d. a cursor appears on the display. The cursor may be, for example, faded in or fixed on the display (for example by printing).

Preferably, the cursor is displayed only during image acquisition.

The position of the cursor from the displayed image on the display is preferably transformed into the overall image and / or preliminary overall image. The transformed position of the cursor in the overall image is preferably stored.

Preferably, the cursor is not located at a fixed position of the display. The cursor may preferably be superimposed on the display in a variable position depending on the representation of the overall picture. Preferably, the cursor can also (and particularly preferably the corresponding overall image) be moved on the display if the display - preferably semi-transparent - menus or other controls are displayed.

Preferably, the data processing device calculates in step c. the image transformation (for example, perspective, affine or only translation) of the images to each other. This can for example be like in US 2011/0285810 described, which is hereby fully incorporated in the disclosure of the present application.

The position of the (provisional) second point that co-determines the (provisional) image area is preferably restricted in comparison to the first point. This restriction can be achieved, for example, by recognizing an object (for example a text line) below the first point and restricting the position of the (provisional) second point to positions within the same object (for example within the same text line).

Preferably, the reference image and at least one other image may be included in the image transformation to obtain an overall image. Preferably, images of the continuous image sequence, in particular the real-time images, are transformed into the coordinates of the reference image in order to be integrated into the overall image or preliminary overall image.

Preferably, the data processing device calculates in step c. starting from the first point, a preliminary complete image, this provisional overall image being displayed completely on the display, assuming a provisional second point as the point in an image of the acquired sequence of images lying below a cursor displayed on the display.

The overall image is preferably composed of images of the image sequence which are described in step c. be detected by the image capture device. This can for example be like in US 2011/0285810 described, which is hereby fully incorporated in the disclosure of the present application.

Preferably, the preliminary overall image or region is centered and / or scaled, and most preferably display-filled, displayed on the display. This makes better use of the display area.

Preferably, an object is detected with the image sequence. The object is preferably planar (and, for example, no general 3D geometry such as vSLAM, no cylinder, or 360 ° panorama). Even if the object is not planar, its planar projection is preferably detected in the image plane of the camera.

Preferably, the image capture device also detects between steps b. and d. a continuous sequence of images displayed on the screen.

• – Benutzeraktion (beispielsweise Tap, Click, Auswahl, Sprachkommando, wie beispielsweise in Schritt b., Erstellen eines Fotos in oder vor Schritt b.),
• – erkanntes Objekt,
• – keine Bewegung, und/oder
• – scharfes Bild.

Preferably, a reference image is determined in the method according to the invention. The definition of the reference image can preferably be triggered by the following events:

• User action (eg, tap, click, select, voice command, such as in step b., Create a photo in or before step b.),
• - recognized object,
• - no movement, and / or
• - clear image.

If, for example, one of these events occurs, an image, particularly preferably a high-resolution image, can preferably be made with the image acquisition device to generate the reference image.

Preferably, the reference image in step b. established. Particularly preferred in step b. generates an image which serves as a reference image. The reference image is preferably an image of the image sequence.

The overall image and / or provisional overall image is preferably composed of the reference image and at least one further subsequent image from the image sequence.

In the reference image, the reference coordinates are preferably determined with a reference point. The reference coordinates are preferably two-dimensional. The reference point is preferably the upper left corner of the reference image. All other points or coordinates of the method according to the invention mentioned in this description are preferably relative coordinates or points to the reference coordinates or to the reference point. The coordinates are preferably given in pixels. The pixels preferably correspond to pixels of the images in the image sequence and / or the reference image.

Preferably, the reference image is included in the calculation of the overall image and / or preliminary overall image.

The determination of a changed perspective of the image acquisition device and / or the conversion of coordinates (for example the transformation of camera coordinates into display coordinates or coordinates of the overall image) is good in the standard work Zisserman / Hartley "Multiple View Geometry in Computer Vision" (Cambridge University Press, 2004) described.

Preferably, when the image is captured by the image capture device, the captured image is corrected for the perspective of the image capture device. This can be done, for example, by the determination of image nodes and / or feature descriptors (see below) and / or block matching and by subsequent image transformation on the basis of a transformation estimate.

In order to better take into account changes in the perspective of the image acquisition device from one image to another image (such as the reference image or the overall image or preliminary overall image) in the image sequence, one can preferably use acceleration and / or position sensors. In addition, the change in the perspective of the image acquisition device can preferably also be estimated and / or corrected by algorithms (for example, algorithms such as Kalman or extended Kalman).

image area

Preferably, the image area is two-dimensional.

For example, all points have coordinates. The coordinates are, for example, two-dimensional. Preferably, in step e. when calculating the image area or when calculating the preliminary image area, the coordinates of the points are taken into account. Preferably, the coordinates are given relative to the reference coordinates.

Preferably, an overall image is composed of the image sequence starting with the first image and ending with the second image. For this purpose, preferably at least two, particularly preferably at least ten, very particularly preferably at least 50 images of the image sequence are combined to form the overall image.

The overall image and / or preliminary overall image is preferably composed of individual images, as shown in FIG US 2011/0285810 A1 described or as it is common nowadays in panoramic functions of digital cameras or smartphones.

Are between step b. and step d. Intermediate points have been detected (see above), the computing device calculates a contour from the first point and the second point and the detected intermediate points. The chronological order of the detection of the aforementioned points is preferably not relevant. The contour preferably defines the image area. The contour can be connected, for example, by direct lines between the aforementioned points or by a Bézier curve.

The contour can preferably be defined by two, three and / or four points. As a result, preferably a quadrangular contour can be defined.

When composing the images of the image sequence into an overall image, preferably no absolute coordinates, but only relative coordinates to the reference coordinates are used.

Preferably feature descriptors are determined by the data processing device in the images of the image sequence. Suitable points for feature descriptors are determined, for example, via feature detectors.

Feature detectors provide points in the image. Feature descriptors are used to describe these points.

The type of feature descriptors may preferably be selected from image sections themselves, SPIN Image, Shape Context, SIFT (Scale invariant feature transform), SURF (Speeded Up Robust Features) and / or GLOH (Gradient location-orientation histogram).

An image node is a pixel to which there is a feature descriptor.

On the basis of the relative displacement of the image nodes from one image to another image of the image sequence, overlapping and non-overlapping regions of the individual images of the image sequence can be determined. Thus, the overall picture can be easily assembled from the individual images.

Preferably, at least 200, most preferably at least 500 feature descriptors and / or image nodes are determined in one of the images from the image sequence. Preferably, in each case at least one subsequent image, particularly preferably in at least ten further subsequent images, the image sequence is determined in each case five to 150, very particularly preferably in each case 50 to 100 feature descriptors and / or image nodes. Preferably, at least twice as many feature descriptors and / or image nodes are determined in an image of the image sequence or the (preliminary) overall image as in a subsequent image.

The feature descriptors are kept, for example, in a descriptor list. In the overall picture or preliminary overall picture, for example, the descriptors of the considered images are adopted. For example, the descriptor list grows incrementally with each additional picture included in the (preliminary) overall picture.

Preferably, the image area is completely within the overall picture.

For example, the (provisional) image area may be completely within the first image or the last image or the reference image.

Preferably, the image area is larger than a single image.

Preferably, the image area makes up to 90%, preferably up to 50%, most preferably up to 30%, of the area of the captured total image. The acquired overall image is preferably composed of at least two of the images of the image sequence that are between the steps b. and d. were recorded.

Preferably, the data processing device recognizes that the (provisional) image area includes regions of the (provisional) overall image that have not yet been captured by the image acquisition device. The result of this recognition is displayed to the user, for example. Thus, these regions can still be detected later by the image acquisition device.

Preferably, the method, in particular step e., Runs independently, particularly preferably without connection to a server.

Preferably, the image area (for example, from the data processing device) after step e. further processed. The further processing may preferably be selected from displaying the number of detected objects, displaying context information on detected objects, saving, sending, passing on to another computer program product and / or image processing.

Preferably, the further processing may include the recognition of the type of image area (for example, email address, text, image). As a result, preferably a respective matching computer program product can be opened (example: a recognized email address opens the email program, a recognized date opens the calendar, a recognized photo opens the image editing program).

The other computer program product may be, for example, an e-mail program, an image processing program or another application program. The transfer or saving can be done, for example, as text or in a specific graphics format such as JPG, PNG, GIF, TIFF, PDF.

After step e. It is also possible to insert a graphic element in the image (for example in the overall image) preferably in the image area. This graphical element can be selected, for example, from a line, an arrow, a speech bubble, a pictogram, a rectangle, and / or an oval. Users can, for example, use smartphones to take very high-resolution photos and provide them directly in one step with graphic elements in order to subsequently share them, for example via social networks. After inserting the graphic element, it is preferable to provide a means for entering text. Thus, for example, text can be entered in a speech bubble or for labeling an arrow. The means for entering text may be, for example, a keyboard, virtual keyboard or speech recognition.

Preferably, the graphical element is already during step c. shown in the preliminary screen area on the display.

In or before step b. It is also possible to capture, preferably, an image from the image acquisition device, into which a graphic element is inserted in the image in step e. determined image area is used. For example, in a photo that was taken before step b. a heart in the size of the face of one of the imaged persons should be inserted by appropriate selection of the image area above the person's head.

Preferably, after inserting the graphic element, the real-time image is displayed on the display with the graphic element, so that the user can, for example, take a photo of the person with a graphic element depicted in the photo with a further actuation of an operating element (for example a photo button or button).

The steps b. to e. can also be performed several times in succession in successive passes. In this case, the resulting overall images and / or preliminary overall images of the passages from the data processing device can be compared with one another and be related to one another by means of the descriptors and / or image nodes.

In the aforementioned multiple passes of steps b. to e. Preferably, the (provisional) overall image, the reference point or the reference image of the first pass is used for all passes. It is therefore preferable not to create a new (preliminary) overall image in subsequent runs.

For example, it is possible that selected detected objects selected in a previous pass can again be deselected in a targeted manner.

Preferably, no spatial and / or three-dimensional coordinates are processed in the method according to the invention.

In a preferred embodiment, before or with step b. the image content captured by the image capture device displayed on the display and the first point and the provisional second point used to display a graphical element on the display in the image area calculated therefrom. On the display, the graphic element can be used before or with step b. captured image, or a real-time image of the image acquisition device or the calculated image areas are represented from these. When operating the control element in step d. can the before or with step b. captured image area are further processed (for example, stored, shipped or further processed) with the graphic element drawn thereon.

display

Preferably, in step c. from the first point (and optionally further points such as the intermediate points) and a preliminary second point, a preliminary image area is calculated by the data processing device and the preliminary image area (preferably centered) is displayed on the display.

In step c. Preferably, a preliminary image area is determined by the first point and at least the provisional second point defined by the cursor.

Preferably, the (provisional) overall image is displayed on the display. The (preliminary) overall picture can preferably be displayed centered. Preferably, the regions of the (preliminary) overall image which have not yet been detected by the image acquisition device are identified.

Preferably, the real-time image of the image acquisition device is displayed on the display.

Preferably, an image section of the (preliminary) overall image or real-time image of the image acquisition device can be displayed enlarged on the display. Preferably, the enlarged image section can be chosen such that the (preliminary) image area is displayed on the display.

In step c. For example, as described above, the transformation between two images (for example, between the real-time image and the overall image) is preferably determined. Preferably, the transformation is manipulated to obtain a virtual image. For example, from the transformation between two images, the scaling factor can be determined, which determines the enlargement or reduction of the (provisional) overall image or real-time image of the image acquisition device on the display. Most preferably, the scaling used for the display may be disproportionate compared to the particular scaling factor. As a result, it is possible with a comparatively small change in the distance of the image acquisition device to achieve a comparatively large change in the presentation size, for example, of the overall image.

Preferably, the position of the cursor may be changed based on the estimated transformation. For example, a translation of the image capture device relative to the reference image may cause an additional shift of the cursor, so that a small movement of the image capture device achieves a large cursor movement (measured in reference coordinates).

Preferably, the size and / or shape of the cursor can be changed depending on the object recognition. For example, if a word or letter is recognized, the height of the cursor can be adjusted to the height of the word or letter. Also, for example, the cursor may change the shape of a crosshair to a box or a text cursor if a word or letter is recognized below the cursor.

The position of the cursor preferably locks on positions of adjacent image contents. The image contents may be, for example, image nodes, recognized objects, letters, words, faces, notes, detected edges / gradients in the image, and / or leaf edges.

Leaf edges within the meaning of the invention are, for example, the edges of printed sheets.

object recognition

Preferably, objects are recognized in images. Preferably, images of the image sequence between steps b. and d., most preferably also already detected in step a., objects in real time.

The type of object may be selected from person, face, vehicle, building, letter, grade and / or barcode. The object recognition is done for example by the Viola-Jones method, the Bag-of-features method or Stroke-Width-Transform method. If an object has been detected, the cursor is preferably adjusted and / or a context menu is shown in the display. Preferably action options can be displayed in this context menu.

If the at least one object is a plurality of letters, the text formed from the letters is preferably recognized by OCR (Optical Character Recognition). The text may preferably be stored and / or further processed.

If the at least one object is a person or a face, then this person can be recognized, for example, via biometric facial recognition. The recognized person can for example be compared with databases or social networks, so that additional information about this person can be displayed on the display from the databases and / or social networks.

If the at least one object is, for example, a vehicle or another product, the model can be determined for this vehicle on the basis of its visual characteristics. For example, the identified model can be synchronized with databases or sales platforms so that additional information about that model can be displayed on the display from the databases and / or sales platforms.

• a) alle erkannten Objekte auf dem Gesamtbild oder dem vorläufigen Gesamtbild,
• b) alle erkannten Objekte innerhalb eines Rechtecks, das durch den ersten und den zweiten Punkt definiert wird, und/oder
• c) alle erkannten Objekte in dem Bildbereich. Markierte Objekte können vorzugsweise auf der Anzeige kenntlich gemacht werden (beispielsweise durch farbliche Hervorhebung).

Preferably, recognized objects can be marked. Preferably, the following objects are selected:

• a) all recognized objects on the overall picture or the preliminary overall picture,
• b) all detected objects within a rectangle defined by the first and second points, and / or
• c) all recognized objects in the image area. Marked objects may preferably be indicated on the display (for example, by color highlighting).

The (marked) objects can preferably be further processed, for example counted. Thus, for example, an object density can be determined. This is especially useful in medical applications such as the counting of liver spots.

Further embodiments

In a further embodiment, the object on which the invention is based is achieved by a system for image acquisition comprising at least one image acquisition device, a display and a data processing device, the system being configured to carry out the method according to one of the preceding claims.

In a further embodiment, the object on which the invention is based is achieved by a computer program product executable on at least one processor which, when running on a data processing device, comprises at least step e. of the method according to the invention.

The method according to the invention can be used, for example, in various applications:
With the method according to the invention, for example, photos can be created that are larger than the camera resolution and, for example, go beyond the resolution of the camera, both in height and in width.

With the method according to the invention, for example, graphics and texts can be recognized on documents. The method according to the invention can also be used, for example, as a camera-based scanner.

Further practical embodiments and advantages of the invention are described below in conjunction with the drawings. Show it:

1 shows the sequence of the method according to the invention in the most general form according to the first independent claim.

2 shows the sequence of a preferred method according to the invention, as described in the example below "make and label photo".

3 shows the structure of the system according to the invention.

4 shows the sequence of the method according to the invention with reference to individual exemplary examples of use.

1 and 2 show a method sequence of the method according to the invention, as described in detail in the embodiments below.

In 2 the image area is not stored until after a predetermined time of eg ten seconds, the touch screen has not been touched again to insert a label.

3 shows as a data processing device 10 a smartphone with keyboard 12 , The data processing device 10 has memory 14 , Flash memory as a storage medium 16 , Firmware 18 all over a data bus 20 with a processor 22 and a transceiver 24 (WLAN, mobile, Bluetooth, ...) are connected. With the data bus 20 are also an ad 26 , the keyboard 12 , a camera 28 and an acceleration sensor 30 connected.

4a. shows step b. the method according to the invention. The recording area 32 is aligned so that the cursor 34 is positioned on the upper left corner of the text area to be captured. The touch screen 26 is touched and thus the control is operated to determine the first point 38 ,

The touch screen will be in 4b. held ( 40 ) and the recording area 32 the camera in step c. the method of the invention changed so that by the changed position of the cursor 34 a preliminary image area 36 arises.

In 4c. the touchscreen is released ( 42 ) when the shooting area 32 changed so that the cursor 34 is positioned at the lower right corner of the text area to be detected and the image area 36 contains all the text to be recorded.

In 4d. became the image area 36 edited and inserted into an e-mail of an e-mail program to send it.

embodiments

Reference example - Image capture with a smartphone: on a standard smartphone such as the Samsung Galaxy S7, an app (a computer program) was installed that could access the touch screen (display, control) and the camera (image capture device) and that was configured, the inventive method as follows:
The image captured by the camera was displayed in real time on the S7 touchscreen. In the middle of the screen a semitransparent cross was shown as a cursor. In front of the camera, a rectangular piece of paper with a short text was placed on a table. The camera of the smartphone was aimed at the paper. The smartphone was held so that the center of the cross pointed to the upper left corner of the paper. Now the touch screen has been touched at any point (in this case, bottom right) and kept touching. So the finger was still on the touchscreen after the first touch. Now the camera's shooting range has been changed by moving the smartphone so that the center point crosses to the bottom right corner of the paper. The finger was removed from the touchscreen. The image area, which resulted from a rectangle between these two points, was calculated by the smartphone as follows:
Touching the touch screen, the current image of the camera was saved as the first image. This image served as a reference image. The upper left corner point of the image was the reference point and determined the reference coordinates (0x / 0y). Since the S7 had a resolution of the camera of 4272 × 2848 pixels and a resolution of the touch screen of 2560 × 1440, the camera image was scaled down accordingly to be displayed on the touch screen. The overall image was the reference image recorded with the camera at the time the touch screen touched for the first time. The center of the cursor's cross was on the touch screen at the position (1280 x / 720 y). This position corresponded to the point (2136 x / 1424 y) on the reference image and thus also the overall image at that time. This point was saved when touching the touch screen and displayed in red on the touch screen to colorize it. When the smartphone was panned to position the bottom right corner below the center of the cursor's cross, the camera images were recorded every 20 seconds for the sequence of images. The provisional overall picture was assembled from the reference picture and the camera pictures of the picture sequence. For this purpose, characteristic points in the image were identified as image nodes and described using descriptors. For this purpose, the method SIFT (Scale invariant feature transform) was used. Feature correspondences were generated in image pairs (current image and preliminary overall image as image pair). From these feature correspondences the transformation was estimated using robust methods (RANSAC). Subsequently, filters were applied (for example, extended Kalman to stabilize the camera movement by software). On the basis of the corresponding features of the images of the image sequence, the preliminary overall image could be assembled from the images of the image sequence. Of course, the overall picture was larger than the first reference picture and was shown in its entirety on the touch screen. The red dot that defined the first point was still displayed at the coordinates (2136 x / 1424 y) of the overall image. So, the red dot wandered to the upper left of the screen as the overall image, looking for the lower right corner of the paper, was extended to the lower right by the other captured images in the image sequence. The cross as a cursor was still displayed on the pixel in the overall image, which corresponded to the center of the camera image - so moved to the bottom right opposite the center. Finally, the center of the cross of the cursor was visible above the lower right corner of the paper. The finger was removed from the touch screen. At the time of releasing the touch screen, the last frame of the sequence was recorded by the camera and assembled with the previously taken pictures as previously described to form the final overall picture. The coordinates of the image taken at that time by the camera corresponding to the center of the cross corresponded to the second point and the coordinates were (5002 x / 3120 y) in the whole image. The smartphone has then out of the Overall picture a rectangle with the two opposite points (2136 x / 1424 y) / (5002 x / 3120 y) cut out as an image area and stored. This image area was then passed on to a text recognition program on the smartphone, which was then able to correctly recognize the text on the paper.

Example - Marking a rectangular area: The procedure was the same as in the first-mentioned reference example, unless the following is done differently. Initially, a cursor was displayed at a reference point in the middle of the display. In order to mark a rectangular image area, the device was initially held so that the cursor points to the upper left corner of the area to be selected (first point). Touching the touch screen (control element, similar to mouse button Down or other event) started the selection. The upper left corner remains fixed in the reference coordinates, ie fixed relative to the image content. The lower right corner of the preliminary and final image area was determined by the cursor. During the selection, the position of the lower right corner changed by pivoting the camera (image acquisition device). By controlling the cursor position, the center of the middle of the screen selection corresponded to: The device was moved to correspond to a shift of (x, y) pixels. In this invention, the cursor has been moved around further (x, y) pixels in the same direction. The rectangular selection had a size (width, height) of (2x, 2y). By releasing the touchscreen (control element, button up), the selection process was completed.

Example - Marking with the lasso effect: The procedure was the same as in the first reference example, unless otherwise stated below. The marking of an image area with a rectangle could also be transferred to the marking by an arbitrarily shaped contour ("lasso"). For this purpose, the geometric center of the selection and its position relative to the reference point (display center) was continuously determined during the selection. If the device (ie the reference point in the display) was shifted by (x, y) towards the center of the selection, then the cursor is shifted by further (x, y) pixels in the same direction. At the beginning of the selection, the cursor is at the reference point. There are also other, non-linear displacements of the cursor as a function of the 2D displacement between the center and reference point conceivable.

Object-based image selection: In this example, the reference example is expanded and explained. In the preliminary overall picture, letters (objects) were recognized by letter recognition methods (for example, stroke-width-transform method and OCR). Image nodes with coordinates and descriptors were saved for these objects. Before releasing the touch screen (ie in step c.), Individual letters could be selected by positioning the center of the cross over a letter by the cursor. The selected letter was colored blue for the length of time that the cursor was positioned over the letter. A semitransparent context box was shown in the display on the edge where the letter "R" was shown, because this letter had just been selected.

In a modification, before step b. Move the cursor over a letter, then press the touchscreen, and then move the cursor over the subsequent letters of the same word (step c.) and release it on the last letter (step d.). In doing so, the cursor changed its shape and height in that the shape was changed in a text cursor that took the height of the recognized word. The letters of the word were colored blue and the recognized word displayed in a semi-transparent box after releasing the touch screen. In this case, the calculated image area from step e. the area in the overall image that took the recognized word.

Making and labeling: In this example, the reference example is expanded and explained. After step e. the cursor was redisplayed in the middle of the screen. Furthermore, the calculated image area on the display was represented by green lines. This could be realized by the fact that the overall image was compared with the real-time image from the camera on the basis of corresponding features, as described in the previous examples. While the real-time image from the camera was displayed on the screen, the edges of the calculated image area on the screen were represented by green lines. Of course, depending on where you looked at the camera, the green borders on the display were moved or even moved out of the display. In this particular example, you could label the image area and thus the photo as you have positioned the center of the cross of the cursor within the displayed image area and touched the touch screen. As a result, a text field was displayed at the selected position, in which you could now enter a label with the virtual keyboard of the smartphone. As soon as the user had finished the labeling by pressing the Enter / Return key, the calculated image area together with the label was saved as an image in JPG format on the smartphone and could be sent by e-mail.

The features of the invention disclosed in the present description, in the drawings and in the claims may be essential both individually and in any desired combinations for the realization of the invention in its various embodiments. The invention is not limited to the described embodiments. It can be varied within the scope of the claims and taking into account the knowledge of the person skilled in the art.

LIST OF REFERENCE NUMBERS

10

 Smartphone

12

 keyboard

14

 random access memory

16

 storage medium

18

 firmware

20

 bus

22

 processor

24

 transceiver

26

 display

28

 camera

30

 accelerometer

32

 reception area

34

 cursor

36

 image area

38

 Operation control element (first point)

40

 Recording area is changed

42

 Operation control element (second point)

44

 Capture the image sequence

46

 Calculation of the image area

48

 Output of the image area

50

Touched control again

52

 Input of text

54

 Termination of input

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• WO 97/48037 A1 [0004]
• US 2010/0031041 A1 [0005]
• US 2011/0285810 A1 [0006, 0060]
• US 2011/0285810 [0040, 0044]

Cited non-patent literature

• Zisserman / Hartley "Multiple View Geometry in Computer Vision" (Cambridge University Press, 2004) [0054]

Claims (11)

A method of image acquisition for execution on a system for image capture comprising at least one image capture device, a display and a data processing device, comprising the following steps: a. the image capture device captures a continuous sequence of images displayed on the display, b. by actuating a control element, a first point is marked on a first image, c. the recording area of the image acquisition device is changed, d. by actuating a second control element, a second point is marked on a second image, and e. the data processing device calculates an image area at least from the first point and the second point. A method according to claim 1, characterized in that in step c. from the first point and a preliminary second point a preliminary image area is calculated by the data processing device and the preliminary image area is centered and / or scaled on the display. A method according to claim 1 or 2, characterized in that the image area is two-dimensional. Method according to one of claims 1 to 3, characterized in that the image area of the area makes up up to 90%, preferably up to 50%, most preferably up to 30%, of the total recorded image. Method according to one of claims 1 to 4, characterized in that in step b. and / or step d. the first and / or second point is that point in the first and / or second image which is arranged under a cursor marked on the display at the time of operation of the respective operating element. A method according to any one of claims 1 to 5, characterized in that in steps a., B. and / or d. Real-time images are displayed on the display. Method according to one of claims 1 to 6, characterized in that the method runs autonomously, preferably without connection to a server. Method according to one of claims 1 to 7, characterized in that the data processing device in step c. starting from the first point, a preliminary total image is calculated, this provisional overall image being displayed completely on the display, assuming a provisional second point as the point in an image of the acquired sequence of images lying below a cursor displayed on the display. Method according to one of claims 1 to 8, characterized in that during steps a. to d. a cursor appears on the display. A system for image acquisition comprising at least one image capture device, a display and a data processing device, wherein the system is configured to carry out the method according to one of the preceding claims. Computer program product executable on at least one processor, which at least expires on a data processing device step e. of the method according to one of claims 1 to 9 performs.

Images