RU2417545C2 - Photographic camera for electronic device - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: digital photographic camera has a support structure, an objective lens held by the support structure and having an optical axis, a sensitive element held by the support structure under the objective lens and having a certain number of adjacent pixel rows, where each pixel row contains a certain number of pixels, and each pixel includes an image sensor, and the image signal processor connected to the sensitive element includes an image scaling device which is configured to scale each pixel row in accordance with the scaling factor which differs from the adjacent pixel row. The image scaling device is configured to correct the oblique angle between the sensitive element of the photographic camera and the objective lens, the image of which is being captured.

EFFECT: avoiding geometrical distortions caused by the position of image capturing apparatus relative the object whose image is being captured.

25 cl, 16 dwg

Description

RELATED APPLICATION

This application claims the priority of U.S. provisional patent application Serial Number 60 / 760,899, entitled "Camera for Electronic Device", filed January 20, 2006, the disclosure of which is hereby incorporated by reference as if it were set forth in full.

FIELD OF THE INVENTION

The present invention relates to a camera for use in a communication device, such as a camera that is part of a radio terminal, for use in video telephony. More specifically, the invention relates to a solution for adjusting a viewing direction of a camera in a communication device having a display.

BACKGROUND OF THE INVENTION

The production of cell phones in the world over the past decades has received tremendous development. From source analog systems, such as those defined by the AMPS (Advanced Mobile Phone System) and NMT (Nordic Mobile Phone) standards, development over the past years has almost exclusively focused on standards for digital solutions for cellular radio network systems such as D-AMPS (e.g. as specified in EIA / TIA-IS-54-B and IS-136) and GSM (Global System for Mobile Communications). Currently, cellular technology is entering the so-called 3rd generation (3G) through communication systems such as WCDMA, which provide several advantages over the above-mentioned 2nd generation digital systems.

A large number of improvements in mobile phone technology relate to functional characteristics, such as better displays, more efficient and longer battery life, and to means for generating polyphonic ring tones. One functional feature that is becoming more and more common is the integrated camera. Some mobile phones currently have cameras with camcorder functionality. With the advent of high-speed data services such as EDGE (the development of the GSM standard with an increased data transfer rate) and 3G, the availability and consumer properties for video transmission services will increase. In particular, recently, mobile video telephony with the simultaneous transmission of sound and moving images has become commercially available.

For stationary use, videoconferencing systems typically include a camera mounted on or near a communication terminal, such as a personal computer (PC), or integrated in a telephone using Internet Protocol (IP). Using such a system can be quite simple, since the user is located in front of the terminal with a camera aimed at the user. However, mobile video conferencing is more difficult. The terminal can be located on a stand on the desktop, from which the camera in the device is directed to the object to be studied, which you want to shoot, usually the user. A simpler way to use a mobile phone for video conferencing with face-to-face transmission is to hold it in such a way that the built-in camera is manually aimed at the user. When communicating via a mobile handheld terminal, the user, therefore, can hold the terminal constantly in front of the face so that the receiving side can see the face of the user, i.e. transmitting side.

The problem related to video conferencing using a radio terminal is due to the fact that the built-in camera is usually located next to or parallel to the display, i.e. the optical axis of the camera is perpendicular to the surface of the display. Therefore, the terminal should be directed at an angle of more or less than 90 ° to the face in order to obtain a proper image of the user. However, many users find this way of holding the terminal uncomfortable. In addition, for most mobile phone designs, it can be difficult to use a terminal located on the desktop without additional fixing means, as this may require that the user's face be above the terminal. A related problem is that the terminal may also include a small flashlight pointing in parallel with the camera to illuminate the subject. When the camera and the flashlight are directed at the user's face at an angle of 90 °, there is a risk that the reflection of the user's face in the display surface will distort the images displayed on the display.

Even when the camera is configured in such a way that it can be held at an angle to the subject being shot, for example, the face of the camera user, there is a problem of distorting the perspective of the image. This can lead to problems when the true representation of the size of the object is crucial. In the case of video telephony, the captured image of the user's face will tend to have a wider lower part than the upper part of the face if the camera is held at an oblique angle to the face.

SUMMARY OF THE INVENTION

The main objective of the invention is to provide a solution for the formation of digital images, where the camera can be held at an oblique angle to the object to capture its image, which usually leads to a distorted image.

According to a first aspect, the claimed purpose is achieved by using a digital camera containing a support structure, a lens held by a support structure and having an optical axis, a sensing element held by a support structure under the lens and containing a number of adjacent pixel rows, where each pixel row contains a number of pixels , and each pixel includes an image sensor, and an image signal processor unit connected to the sensing element, including a device An image scaling property configured to scale each row of pixels according to a scaling factor that differs from an adjacent row of pixels.

In an embodiment, the image scaling device is configured to scale each row of pixels according to a scaling factor having a value proportional to the position of the line between the start line and the end line.

In an embodiment, the image scaling device is configured to be responsive to input of a scaling factor of the start line and a scaling factor of the end line, and comprises a calculation function configured to calculate scaling factors for each line between the start line and the end line.

In an embodiment, the image scaling device is configured to calculate an input row length for a row of pixels as a ratio between a desired output row length common to all pixel rows and a scaling factor for that row; and

Configurable to scale the image signals detected by the pixels of this line and which are within the input line length of the pixels to the desired output line length.

In an embodiment, the image scaling device is configured to create an output image with centered lines.

In an embodiment, the image scaling device is configured to calculate a central starting point for each input line from the sensor using the formula:

,

,

where start _n is the first pixel to be processed in line n; l is the number of pixels in the entire line; and l _n is the number of pixels to process in row n.

In an embodiment, the camera module is constituted by a support structure where an image signal processor is included in the camera module.

In an embodiment, the image scaling device is configured to determine a position in a predetermined image format of the output pixel of a certain pixel row, to determine the corresponding position in the detected image by back-scaling using the scaling factor for said certain row, and to determine the brightness value for the output pixel by interpolating brightness values that are detected by pixels adjacent to the mentioned th corresponding position in the detected image.

In an embodiment, the image scaling device is configured to calculate scaling factors depending on a given estimated oblique angle between the image plane of the sensor and the object whose image you want to capture (capture).

In an embodiment, the camera’s field of view is determined by the working area of the surface of the sensor, which is offset from the center relative to the optical axis of the lens.

In an embodiment, the image scaling device is configured to calculate scaling factors S _n for each row n by a function S _n = m + n * k, where m and k are constants.

According to a second aspect, the claimed purpose is accomplished by means of a communication device comprising a housing; and a digital camera module including a support structure, a lens held by the support structure and having an optical axis, a sensor element held by the support structure under the lens, containing a number of adjacent pixel rows, where each pixel row contains a number of pixels, and each pixel includes an image sensor, and an image signal processor unit connected to the sensing element, including an image scaling device, is configured annoe to scale each row of pixels according to the scaling factor that is different from an adjacent row of pixels.

In one embodiment, the communication device comprises a radio signal transceiver and a control unit configured to provide a scaled video signal from the digital camera module to the radio transceiver.

In one embodiment, the communication device comprises a display configured to represent the scaled image that is provided by the digital camera module.

According to a third aspect, the claimed purpose is achieved by a method for capturing an image using a digital camera, comprising the steps of:

camera directions to the subject;

detecting image signals in a sensor containing a number of adjacent rows of pixels, where each row of pixels contains a number of pixels, and each pixel includes an image sensor;

processing the detected image signals by scaling each row of pixels according to a scaling factor that differs from the adjacent row of pixels to provide a scaled image; and

output scaled image.

In one embodiment, the method comprises the step of:

scaling each row of pixels according to a scaling factor having a value proportional to the position of the row between the start row and the end row.

In one embodiment, the method comprises the steps of:

setting the scaling factor of the start line and the scaling factor of the end line; and

calculating scaling factors for each line between the start line and the end line.

In one embodiment, the method comprises the steps of:

calculating the input row length for the row of pixels as the ratio between the desired output row length common to all rows of pixels and the scaling factor for this row;

and

scaling image signals detected by the pixels of this line, and which are within the input line length of the pixels, to the desired output line length.

In one embodiment, the method comprises the step of:

providing a scaled image with centered lines.

In one embodiment, the method comprises the step of:

calculating the central starting point for each input line from the sensor using the formula:

,

,

where start _n is the first pixel to be processed in line n; l is the number of pixels in the entire line; and l _n is the number of pixels to process in row n.

In one embodiment, the method comprises the step of:

processing the detected image by the image signal processor, made as a unit with a digital camera in the camera module of the communication device.

In one embodiment, the method comprises the step of:

transmitting the scaled image to a remote receiver using a radio transceiver in a radio terminal.

In one embodiment, the method comprises the step of:

Representation of the scaled image on the display.

In one embodiment, the method comprises the steps of:

setting the image format;

determining the position of the output pixel in a row of pixels in the image format;

determining an appropriate position in the detected image by backscaling using the scaling factor for said row; and

determining a brightness value for the output pixel by interpolating the brightness values that are detected by pixels adjacent to said corresponding position in the detected image.

In one embodiment, the method comprises the step of:

calculating scaling factors depending on a predetermined estimated oblique angle between the image plane of the sensor and the object whose image you want to capture (capture).

BRIEF DESCRIPTION OF THE DRAWINGS

The features and advantages of the present invention will become more apparent from the following description of preferred embodiments with reference to the accompanying drawings, in which:

1A and 1B schematically illustrate a handheld radio communication terminal including a digital camera and a display, according to some embodiments of the invention;

figure 2 illustrates the terminal of figure 1 when used for video conferencing, according to some variants of the invention;

figure 3 schematically illustrates how the camera of the terminal is held at an angle to the face of the user;

4 schematically illustrates a digital camera module, according to some embodiments of the invention;

5 schematically illustrates a conventional camera phone;

6 schematically illustrates some aspects of a camera phone, according to some embodiments of the invention;

7 schematically illustrates some aspects of a camera phone, according to additional embodiments of the invention;

FIGS. 8 and 9 schematically illustrate a digital camera module, according to some embodiments of the invention;

10 and 11 schematically illustrate a digital camera module, according to additional embodiments of the invention;

12 schematically illustrates a distorted image caused by the camera being held at an angle to a rectangular object;

13 and 14 schematically illustrate a distorted image and a corrected image according to an embodiment of the invention;

Fig. 15 schematically illustrates an image obtained from a rectangular object held at an oblique angle to the surface of a camera sensing element.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Embodiments of the present invention will be described below in more detail with reference to the accompanying drawings, in which embodiments of the invention are shown. This invention, however, can be implemented in many different forms and should not be interpreted as limited to the embodiments set forth herein. More specifically, these embodiments are provided so that this disclosure of the invention is comprehensive and complete and fully conveys the scope of the invention to those skilled in the art. Throughout the text, the same numbers refer to the same elements.

Although the terms are “first,” “second,” etc. can be used in this document to describe various elements, it should be understood that these elements should not be limited to these terms. These terms are used only to distinguish one element from another. For example, the first element could be called the second element, and likewise, the second element could be called the first element, without departing from the scope of the present invention. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed terms.

The terminology used in this document is intended only for the purpose of describing particular embodiments and is not intended to limit the invention. When used in this document, the singular is also intended to include the plural, unless the context clearly shows otherwise. Additionally, it will be understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” as used herein, indicate the presence of declared features, numbers, steps, actions, elements, and / or components, but do not interfere with the presence or addition of one or more signs, numbers, steps, actions, elements, components and / or their groups.

Unless otherwise specified, all terms (including technical and scientific terms) used in this document have the same meaning, which in most cases is understood by an ordinary specialist in the field of technology to which the invention relates. Additionally, it will be understood that the terms used in this document should be interpreted as having a meaning that is consistent with their meaning in the context of this description and in the relevant field of technology, and should not be interpreted in an idealized or too formal sense, unless so explicitly defined in this document.

The present description relates to the field of electronic devices, namely, communication devices including a camera and a display for displaying photographs taken by the camera, which are configured so that the user can look at the display while the camera is aimed at the same user. Some embodiments of the invention relate to a communication terminal configured for video telephony. Such a communication terminal may be, for example, a DECT telephone (European Digital Cordless Telephone) connected by a wire to a PSTN (Public Switched Telephone Network) socket, or an IP telephone with a housing including a display and a camera. In some embodiments, the communication terminal is a radio communication terminal, for example a mobile telephone, operating to exchange information via a radio base station and / or directly to another radio terminal.

Embodiments will now be described with reference to the accompanying drawings.

1A illustrates an electronic device in the form of a portable communication terminal 10, for example a mobile phone, according to some embodiments of the invention. Terminal 10 includes a support structure 11 including a housing, and a user interface including a keypad or keyboard 12 and a display 13. Terminal 10 may also include an audio interface including a microphone and speaker, a radio transmitter circuit, an antenna, a battery, and a microprocessor system, including appropriate software and memory for storing data for radio communications, all held by the supporting structure 11 and enclosed in a housing. In addition to these elements, terminal 10 also includes a digital camera 14, the aperture of which is indicated in FIG. 1A.

As shown in FIG. 1A, the aperture of the camera 14 and the display 13 can be placed so that both are visible from the normal viewing position. For example, the aperture of the camera 14 and the display 13 can be placed on one side of the housing 11. Thus, the camera 14 can be directed to the user while the user is looking at the display 13. Accordingly, the communication terminal 10 can be used for video telephony.

Embodiments of the present invention may be further described with reference to a schematic drawing of the communication terminal 10 shown in FIG. 1B. Referring to FIG. 1B, an exemplary communication terminal 10 in accordance with some embodiments of the present invention includes a keypad 12, a display 13, a transceiver 26, a storage device 16, a microphone 15, a speaker 19, and a camera 14 that communicate with a control unit or the processor 20. The transceiver 26 typically includes a transmitter circuit 27, a receiver circuit 28, and a modem 29 that cooperate to transmit and receive radio frequency signals to remote transceivers via an antenna 25. Radio frequency the signals transmitted between the communication terminal 10 and the remote transceivers may contain traffic signals and control signals (eg, paging / message signals for incoming calls) that are used to establish and maintain communication with the other party or destination.

The storage device 16 may be a general-purpose storage device that is used both to store program control commands for the processor 20, and data, for example, audio data, video data, configuration data and / or other data that the processor 20 can access and / or use . The storage device 16 may include non-volatile memory with online write and read, read only memory and / or non-volatile memory.

Referring to FIG. 2, the use of a communication terminal 10 for video telephony is illustrated. Typically, in a video telephony session, an image 21 of the remote participant is transmitted to the terminal 10 and shown on the display 13 in the designated window 22. At the same time, a thumbnail image 23 of the user of the terminal 10 captured by the camera 14 can also be displayed on the display 13 in the window 24. Window 24 can be displayed in a separate window / display of the terminal 10 and / or in the subwindow of the display 13 as a picture-in-picture. Thus, the user can receive visual feedback on how the camera 14 is directed, and can control the terminal 10 for correct guidance.

A problem related to video telephony has already been described, namely, that it may be more comfortable to hold the terminal 10 at a certain angle to the user 30, indicated by θ in FIG. 3, and not parallel to the face of the user. The location of the terminal 10 at an oblique angle to the object to be displayed, usually the face of the user, can also facilitate the use of the terminal 10 for video telephony by placing the terminal on a support, such as a work surface. However, deviation of the conventional terminal may result in the effect of shifting the captured image, as the camera may not be oriented correctly. The offset of the captured image in the designated image window will be visible on the display and, of course, for the remote participant receiving the captured image. With increasing angle θ, the user's face may disappear in the window. At some point, the face will fall out of the field of view of the camera, which usually happens in the range of 50-70 ° full angle. Referring to FIGS. 1-14, various embodiments of a camera and a communication device in the form of a terminal including a camera that are configured to take an image at an oblique angle as described in FIG. 3 will now be described. More specifically, a camera configured to capture (capture) an image from a main line of sight that is inclined relative to the optical axis of the camera will be described first. In addition, a camera and method for correcting or adjusting the perspective of a captured image that is distorted due to angle deviation will be described.

4 schematically illustrates a digital camera module 14 for use in an electronic device, such as a terminal 10, according to some embodiments of the invention. The camera module 14 includes a lens 41 including one or more separate lenses made, for example, of plastic or glass and having an optical axis 45 indicated by a dot-dash line. The sensing element 42 with the upper surface 43 of the sensing element is located at a distance and parallel to the lens 41 by the support element 44, which may include a sealed plastic housing. The camera module may also include an image signal processor (ISP) 46, which may be connected to the back of the sensor 42. Alternatively, the ISP 46 may be connected to the sensor 42 by a wire, such as a flexible cable. The geometry of the camera module 14, including the focal length and aperture of the lens 41 and the image plane size defined by the surface 43 of the sensor and its position relative to the lens 41, determines the field of view of the camera module 14. For the purpose of clearly describing the invention, the term "main line of sight" will be used to designate the main beam through the lens 41 to the center of the image area used. Typically, the surface 43 of the sensor is located centrally under the lens 41, and the main line of sight of the camera 14, therefore, coincides with the optical axis 45. The surface 43 of the sensor, as a rule, can be rectangular or even square and can be symmetrical about the optical axis 45.

Fig. 5 illustrates a conventional terminal 10 including a camera 14 and a display 13. In Fig. 5, the axis 51 indicates the normal direction of the display 13, i.e. an axis perpendicular to the surface of the display 13. In addition, the optical axis 45, which represents the main line of sight for the camera 14, is indicated as parallel to the normal direction 51.

6-7 illustrate terminals 10A, 10B in accordance with some embodiments of the invention in a simplified side view, which illustrates only the display 23 and the camera 14. FIG. 6 illustrates a terminal 10A in which the camera 14 is tilted so that its optical the axis and the main line of sight 45 inclined relative to the direction 51 of the normal to an acute angle φ. In some embodiments, the implementation of the camera 14 may be tilted at an angle φ, which corresponds to a given working angle θ, which is indicated in figure 3. Thus, terminal 10A can be used for video telephony when it is held at an angle to the user without shifting the captured image. However, in these embodiments, since the camera 14 may tilt, it may take up more space in the terminal 10A. In addition, securing the camera 14, for example, to a PCB (printed circuit board) in an oblique orientation in terminal 10A, may require an additional installation device, such as an intermediate wedge-shaped element.

7 illustrates a terminal 10B, according to additional embodiments of the invention, at which position 51 indicates the normal direction of the display 13, i.e. an axis perpendicular to the surface of the display 13. In addition, the optical axis 45 of the camera 71 is indicated as parallel to the normal direction 51. The camera 71 may be mounted inside the terminal 10B parallel to the display 13, for example, by soldering or other type of connection and connection to a common PCB. However, the field of view of the camera 71 having the main line of sight 72 is set by the working area of the surface of the sensor 43, which can be offset relative to the center with respect to the optical axis 45 of the lens (see Fig. 4). The working area may be the entire area of the surface of the sensor 43, where the entire surface of the sensor is offset from the center. Alternatively, the working area may be offset from the center by a part of the otherwise centered surface 43 of the sensor element, in which case the camera 71 may be basically similar to the camera 14 based on the elements shown. The difference is which pixels on the surface of the sensor are used to read the image.

Figs. 8 and 9 schematically illustrate certain aspects of a chamber 71, according to some embodiments of the invention. The reference numerals used in FIG. 4 are also used in FIGS. 8 and 9 for the respective elements. ISP 46 is excluded from FIGS. 8 and 9 because it does not need to be mechanically attached directly to the camera module 71. Fig. 8 is a side view of the chamber 71, and Fig. 9 is a perspective view of the chamber 71, in which, for simplicity, the supporting element 44 (supporting structure) is excluded.

The sensor element 42 may include an image sensor having a full-sized sensor surface 43 within the area specified by the length A and width C, and may include a number of pixels, for example, 400 × 400, 640 × 480, or any other matrix layout . In this embodiment, the working area 91 of the sensor surface 43 is still defined, which includes only a subset of the total number of pixels. In the illustrated example, the region 91 (the shaded region) is rectangular and has a length B <A and a width D <= C. In addition, the working area 91 may be offset from the center with respect to the surface 43 of the sensor, where the optical axis 45 of the lens 41 intersects the surface 43 of the sensor. It is possible to set the region 91 as offset from the center along the x and y axes indicated in FIG. 9. However, in the illustrated embodiment, the region 91 is offset from the center only along the x axis and centered on the y axis. The working area 91 may be offset from the center to the right side face along the x axis and may occupy all the pixels behind the right side face, but not all pixels towards the left side face of the sensor surface 43. Alternatively, the work area may be less offset from the center and may not include the outermost pixels on the right side of the sensor surface 43. Along the axis y, the working area 91 may be narrower than D, than the entire width C of the surface 43 of the sensor, as illustrated in the drawing.

The center of the working area 91 can be the center of reading the image plane, and the main line of sight 72 can be set from the center of the working area 91 and the optical center of the lens 41. This main line of sight can pass at an acute angle φ to the optical axis 45, where the value of φ may depend from the distance between the center of the working area 91 and the optical axis 45. Being sharp, the angle φ is by definition greater than 0 ° and less than 90 °. However, for practical purposes, the angle may be in the range of 5-20 ° or even 5-10 °.

As an example, the surface 43 of the sensing element may include a 400 × 400 pixel array of image sensors. However, for the purpose of video conferencing, this may be an excessive number of pixels. QCIF (Quarter of a Common Intermediate Format) is a video conferencing format that specifies data rates of 30 frames per second (fps), with each frame containing 144 lines and 176 pixels per line. This is one quarter of the resolution of the full CIF, which defines 355 × 288 pixels. QCIF support is required by the ITU H.261 video conferencing standard, and essentially only a 176 × 144 pixel matrix is required. This is less than half the available number of pixels in each direction. In order to improve image quality, it is possible to use twice as many lines with twice as many pixels per line, that is, CIF, which is suitable for a 400 × 400 matrix.

In some embodiments, a workspace 91 including 355 × 288 pixels is defined on the surface 43 of the sensor including 400 × 400 pixels, extending inside from one side face of the centered surface 43 of the sensor, and is centered along this side, as shown in Fig.9. The sensor 42 of 3.2 × 3.2 mm in size with a pixel density of 3.6 micrometers has a surface (A × C) of the sensor of approximately 1.44 × 1.44 mm, and the work area will have a length B of 288 / 400 × 1.44 = 1.037 mm. The center of the working area can then be located 1.44 / 2-1.037 / 2 = 0.2 mm from the center of the surface 43 of the sensing element. Assuming that the lens 41 is located at a height of 1.5 mm from the surface 43 of the sensing element, the main line of sight 72 will then have an angle of approximately φ = arctan (0.2 / 1.5), approximately 7.6 ° to the optical axis 45. Using only the QCIF matrix, the corresponding angle would be arctan (1.44 * (1-144 / 400) / (2 * 1.5)), or approximately 17.1 °. However, even if it is necessary to use a QCIF image, it is possible to use the full CIF image plane to improve image quality.

It is also possible to set the work area by means of inverse calculation. For example, suppose that the camera will be used at a certain angle θ, for example 10 °, and it is necessary to use a QCIF matrix. The required offset Δ of the center of the working area 91, when the distance to the lens is 1.5 mm, then is Δ = 1.5tan (10 °), or approximately 0.26 mm. For such a configuration, the working area 91 will not go out all the time to the edge of the surface 43 of the sensing element; rather, there will be about 55 unused pixel rows at the top of the sensor surface 43.

It should be understood that the above values are given only as possible examples, whereas the provision of a sensor surface offset relative to the center of the working area to obtain a field of view with a main line of sight that is inclined with respect to the optical axis of the camera lens can be applied to any camera geometry, as schematically illustrated in FIG. 4. The sensor 42 may be, for example, a CMOS sensor or a CCD sensor, and may be black and white, gray tones, or a color image receiver. In addition, the work area 91 may be indicated in the equipment or firmware for the ISP camera as offset from the center of the area of interest or window to be considered. In some embodiments, the size and position of the work area 91 can be set as the default value, and thus are always used until user instructions are given to change this setting, for example, via input interface 12.

FIG. 10 illustrates embodiments that may be provided as alternatives to the embodiments described with reference to FIGS. 8 and 9, or that may be combined with the embodiments of FIGS. 8 and 9. Similar to the camera 14 illustrated in FIG. 4, the camera 101 of FIG. 10 includes a lens 41 and a sensor 42 with a sensor surface 43 suspended in parallel and spaced from the lens 41 by a support element 44, and a potentially attached ISP (not shown). An optical axis 45 is defined for the lens 41. The surface 43 of the sensor has a length A. However, in this case, the sensor 42 may not be centered below the lens 41. Instead, the sensor 42 is offset to the side relative to the lens 41, so that the optical axis 45 of the lens 41 is offset relative to the center with respect to the surface 43 of the sensing element. 10, this is illustrated by a sensing element 42 biased to the side in the supporting element 44. Alternatively, the lens 41 may be biased to the side in the supporting element 44.

Using this feature, the main field of view line of sight 102 for the camera 101, extending from the center of the sensor surface 43, can extend at an acute angle φ from the optical axis 45, where the angle φ is proportional to arctan (ΔА / h), where ΔА is the relative transverse offset and h is the distance between the lens 41 and the surface 43 of the sensing element. As an example, if the lateral displacement ΔA of the sensor 42 is 0.2 mm, as indicated in FIG. 10, and the distance between the lens 41 and the sensor surface 43 is 1.5 mm, then the main line of sight will have an angle φ of approximately , 7.6 °.

As indicated, it is possible to combine the embodiments of FIGS. 8 and 9 with the embodiments of FIG. 10, thereby setting the working area on the right side of the sensor 42, and where the sensor 42 is also laterally shifted to the right relative to the lens 41. As an example, consider an example shown for the embodiment of FIG. 9 with a sensor surface 43 of 400 × 400 pixels with a pixel pitch of 3.6 micrometers located 1.5 mm below the lens 41, and also the lateral displacement ΔA of the sensitive element is applied ment 42, equal to 0.2 mm, as shown in Fig.10.

For the CIF embodiment, the work area 91 may extend 288 lines inside from the lateral edge of the sensor surface 43, which is farthest from the optical axis 45. Then the center of the work area is 1.44 / 2-1.037 / 2 + 0.2 = 0 , 4 mm from the optical axis, which means an angle to the optical axis 45 of about α = arctan (0.4 / 1.5), or approximately 15 °. For a QCIF matrix, you can have even a larger angle, or alternatively use a CIF image and scale it to QCIF.

For embodiments in which the sensor surface 43 is offset to the side relative to the optical axis 45, an advanced camera can be applied by adapting each pixel to this center-offset optical geometry. 11 schematically illustrates some elements of a chamber 110, according to some embodiments of the invention. 11 illustrates a camera lens 41 and three pixels 110, 120, 130 of the sensor 42. A vertical dashed line 114 is shown between the pixels 110 and 120, indicating the center of the sensor surface 43 of the sensor 42, while the optical axis 45 intersects the center of the lens 41 Typically, the center of the surface of the sensor 43 and the optical axis 45 may be the same, but according to the embodiments described with reference to FIG. 10, they can be spaced ΔA.

To direct the incoming radiation correctly to the sensing elements of the sensing element 42, each pixel may include a sensor including a photosensitive element 111, 121, 131, for example a photodiode, and microcapacitor lenses 112, 122, 132. The use of microlenses as part of an image sensor is conventional technology to improve the performance of the sensor, as shown, for example, in US patent No. 5,251,038. Accordingly, each pixel of the sensing element 42 may include microcapacitor lenses on the surface of the sensing element to direct light rays to the sensing element.

The design and location of the microlenses may depend on the angle of the main beam in the beam of light incident on the sensor. The angle may vary with the height of the image, i.e. the distance from the center of the optical axis 45 of the camera lens 41. Usually, the farther the sensor is located from the optical axis 45, the shorter the focal length of the condenser lenses should be. In a typical configuration, the focal length of the microlenses increases with distance from the center of the sensor surface 43, and for lenses 122, the focal length may trigonometrically depend on the distance F to the center 114 of the sensor surface 43. However, in the case of the embodiments of FIG. 10 with an offset ΔA, the microlenses can be adapted to an optical center, which is still defined by the optical axis 45, which is no longer the center of the surface 43 of the sensor element. Therefore, in some embodiments of the invention, the estimated optical center 45 is determined for the sensor surface 43, which may not coincide with the physical center 114 of the sensor surface 43 and which will be the actual optical center when combined with the lens 41.

The microlenses of each pixel of the sensor surface 43 can be carefully designed with respect to a given optical center, usually with increasing focal length as a function of increasing distance to the optical center. The focal length for lenses 122 may trigonometrically depend on the distance E = (F + ΔA) to the optical center, that is, the optical axis 45. The specific ratios depend on the overall configuration of the camera and the position to be taken into account, well-known to the person skilled in the art.

As illustrated in FIG. 3, the use of, for example, a mobile phone for video conferencing often occurs at a slight angle to the face of the user. However, a distorted perspective will result from a beveled angle between the user's face and the optical axis of the camera. Moreover, this drawback is applicable even when the camera is configured to capture an image in a field of view having a main line of sight that is at an angle to the optical axis of the camera, for example, in the embodiments described with reference to FIGS. Since the camera is not held parallel to the user's face, the face in the lower region is displayed wider than in the upper region. The image in FIG. 12 clearly illustrates this effect when the user holds a rectangular sheet of paper in front of the face while taking an image using the camera configured according to the embodiment of FIGS. 7-9 with the image area offset from the center. This is often called keystone distortion.

In traditional professional photography, this keystone distortion can be avoided by using special optics, such as lenses with offset and tilt, i.e. with perspective control, or specially designed cameras with perspective control capabilities. In the field of digital image processing, perspective adjustment is a common possibility. Tools such as Adobe® Photoshop provide this capability. Such capabilities can also be used in electronic video conferencing devices, such as mobile phones with integrated cameras, for post-processing of images. However, this perspective adjustment is highly dependent on computing capabilities. This is especially a problem with moving images, i.e. video, with a lot of frames per second for processing. Post-processing, therefore, is not applicable for implementation in a system where the central processor 20 of a mobile phone is used simultaneously for encoding video and other tasks.

It is also advisable to process an image with a higher resolution and shooting angle than the final image in order to improve image quality. This necessitates the transfer of a larger image, i.e. more data, from the camera to the host system. The host system must also process a non-standard image size and convert it to the specified format.

One embodiment includes the ability to correct perspectives using an image pipeline and a camera image signal processor (ISP). There are several advantages with this implementation, which will be explained later. It is also desirable to use an image sensor and optics, which provide a slightly larger viewing angle than expected from the final image.

As can be seen from Fig, the image looks too wide in the lower part compared with the upper part. To fix this, the principle of perspective adjustment is implemented, where the lower rows of pixels are compressed. We believe that a slight compression is applied to each row of pixels as we move up in the image. As a result, the top row is compressed to the least degree, or not compressed at all. The created image at the bottom will be narrower than at the top. To avoid this, you need to crop the image, or as an alternative to start using an oversized image. In the second case, the perspective adjustment process works on a longer line of image data in the lower part than in the upper part. The result is an output image from an ISP with right angles.

The preferred implementation of the perspective correction mechanism is determined by the camera hardware or firmware. A typical digital camera module for a mobile camera with a built-in ISP has a zoom function. The scaling function can be implemented as a digital hardware unit or as computer code executed by a processor, or as a combination of hardware and computer code. However, as mentioned earlier, it is not necessary for the ISP module to be integrated; it can very well be connected by cable to the supporting structure or housing of the camera module. In this sense, the digital camera includes an ISP module that includes a processor and associated ISP software. A conventional zoom device for horizontal and vertical scaling of the image can be installed. It can be configured to independently scale two dimensions. Thus, an image can only be compressed in one dimension, leaving the other untouched. The scaling device can also be configured to scale the image with a factor of n, where n is a floating point number, for example, 1: 1,2, etc. In a preferred embodiment, the individual rows are scaled using an interpolation algorithm, such as linear interpolation, to determine the magnitude of the signal to output for a certain pixel, using the magnitudes of the signals found in two adjacent pixels of the row in question.

According to a preferred embodiment, each image line is scaled by a scaling factor different from the previous and next line. Preferably, the ISP calculates the scaling factors for each row from the input value of the starting and ending scaling factors, for example, the scaling factors of the first and last row. Scaling factors can be expressed as a ratio given by the input line length and the given output line length, expressed in the number of pixels.

In a preferred embodiment, fixed scaling factors are used, since the use of video telephony is clearly defined, and the angle θ between the user's face and the telephone can be estimated with sufficient accuracy. Since the user is looking at the display, which shows his or her image taken by the camera of the electronic device, the user will automatically hold the electronic device so that the face image is centered on the display almost vertically.

Another important property of the scaling device to be imagined is line centering. The image is therefore scaled, preferably, in such a way that the center vertical line through the entire input image is stored in the output image. This can be achieved by calculating the starting point for each input line. Pixels before the starting point of each row are discarded as trailing pixels. In one embodiment, the starting point of each row is calculated from the following equation:

,

,

where start _n is the first pixel to be processed in line n; l is the number of pixels in the entire line; and l _n is the number of pixels to process in row n.

A scaling device that is designed to scale a vertical measurement depends on an information storage unit that stores two or more lines of image data. A scaling device that is designed only to scale horizontal measurements will only need an information storage unit that stores a small number of pixels or, in extreme cases, the entire row of image data. Thus, in order to make an economical sample, a complete scaling device for both measurements is not necessary if vertical scaling is not necessary for other purposes.

To achieve good image quality, the sensor can be designed with at least four times the resolution required in the output image, i.e. with at least twice the number of pixels in both x and y dimensions. An example is the use of the previously mentioned 400 × 400 sensor for an output image of QCIF format. In this case, the vertical scaling device can be simplified since it only needs two rows of image data for vertical scaling.

FIG. 13 illustrates a user image captured by the user with an oblique angle θ between the optical axis of the camera and the user's face corresponding to FIG. Without even knowing the person, it can be noted that the user's chin seems to be wider than in reality, compared with the forehead, since the entire image is tilted.

However, in the image of Fig. 14, the distortion effect from the inclination of the viewing angle is corrected in accordance with the invention by scaling each row or line of pixels sequentially to a level corresponding to the oblique angle θ. Thus, the perspective is true even if the image of FIG. 14 is tilted.

In an embodiment of the invention, the image of a rectangular object of known proportions, held at an oblique angle to the sensor element of the camera, can be used to calculate and adjust the scaling factors. This setting of scaling factors can be done in production and then used as the default setting. Alternatively, the user may be able to run the installation algorithm by using the keypad 12 in the device 10, based on which a rectangular object is held in front of and parallel to the user's face. It is preferable that the installation algorithm prompts the user to hold an object of known proportions, for example, an A4 sheet or letter ("letter"), and confirm with the keypad 12 which type of object is being used. The camera then switches to taking an image of the object, and the processor 20 starts the loop detection software for identifying the image of the object, as shown in FIG. Regardless of the calculation and adjustment of the scaling factors at the factory or after the sale by the user, additional calculations are necessary after the image contour is determined, as described below with reference to FIG.

15 illustrates an image of a sheet of A4 paper. In this exemplary embodiment, the sensor element of the camera has 400 × 400 pixels, which means that there is space for a CIF image of 355 × 288 pixels. However, the output image that you want to create is a QCIF of 176 × 144 pixels, but instead of using only part of the surface of the QCIF size sensor, the CIF image is read and scaled down by a factor of 2 in height and width to QCIF, to get the best image quality. Sheet A4 has a height that is equal to the square root of twice the width of the sheet. On Fig shows the image of the sheet A4 with a height of b lines of pixels, starting with lines d above the initial line for the image CIF. In addition, the image occupies pixels along the bottom and pixels along the top. To calculate the scaling factors, a pair of constants must first be defined:

;

;

;

;

;

;

m = j-kd.

For row n, the scaling factor S _n will then be equal to:

S _n = m + nk.

In a preferred embodiment, only pixels that will participate in the QCIF image are read and scaled to minimize computation, which is especially advantageous for acquiring video images. In such an embodiment, the predetermined output length of each row is 176 pixels. This means that the length L _{n of the} string n to be scaled is:

.

.

As an example, suppose that the following values were found in the image of FIG. 15, calculated in the number of rows for b and d and the number of pixels per row for a and c:

a = 150;

b = 255;

c = 200;

d = 5.

Using the above formulas, we obtain the following result:

S ₀ = 0.448;

S ₂₈₇ = 0.616;

L ₀ = 393;

L ₂₈₇ = 285.

For verification, we can calculate the scaled width a 'of the top edge of the sheet and the scaled width c' of the bottom edge of the sheet, which will be equal to:

a '= a * S ₂₆₀ = 90;

c '= c * S ₅ = 90;

hence a rectangular image. So, the scaling factors for each row n are thus calculated and set, and the number of pixels for processing to a scaled image for each row is determined for the image acquisition scenario where oblique angle θ is used, as in FIG. 15. Preferably, the oblique angle is not measured or detected when the device is used, instead, some oblique angle θ is set as the expected oblique angle for use when the user is working with the device. Typically, the oblique angle may be less than 20 °, for example 10 °.

For any subsequent image captured by camera 14, each line n must be scaled to 176 target pixels, which is 2 × 88 pixels symmetrically with respect to the vertical center axis. For row n, the first target pixel is 88 pixels from the center axis, and the corresponding position in the detected image is therefore 88 / S _n . This position may not be absolutely one separate pixel of the surface of the sensing element from which the value of the image signal can be extracted. Instead, the value of the image signal for this position is preferably interpolated from neighboring pixels by any known pattern. Preferably, the brightness level values and color codes are interpolated separately. The obtained value (s) of the image signal is then assigned to the first target pixel in the output image. The next target pixel that needs to be assigned the value of the image signal is 87 / S _n , and so on, until the vertical center axis is reached. The other side of the central axis is processed accordingly, since the image is scaled symmetrically about this axis. The scaling process is then repeated line by line until all 288 lines of the target image are processed.

By performing these steps using the image signal processor in the camera, the computing power is saved, and the digital signal processor 20 in the device 10 can essentially be used for other purposes. The present embodiment adapted to correct image perspective distortion differs from the previously proposed solutions, which are based on post-processing of images, in that this proposed solution proposes to include processing in the camera / ISP image pipeline. The described sample will correct the perspective immediately, without affecting the host processor that is multitasking. This makes the invention particularly suitable for handheld devices, such as mobile phones with an integrated camera, where light weight and compact size are essential market requirements. Preferred embodiments of the proposed sample further differ from common solutions of scaling devices in that each row of data of an image element can be scaled with a different coefficient than other rows in the image array. They can also be designed without any or very small additional equipment, such as logic circuits, but especially without expensive line buffers.

The sample is also specific in that the zoom device can center the image automatically, which is preferred in video conferencing applications. The proposed solution preferably uses a fixed setting to correct the perspective, since the use of video telephony is well defined.

In the drawings and in the description of the invention, typical embodiments of the invention are disclosed, and although specific terms are used, they are used only in a general and descriptive sense, and not for the purpose of limiting the scope of the invention set forth in the following claims.

Claims (25)

1. A digital camera containing:
supporting structure (44);
a lens (41) held by the support structure (44) and having an optical axis (45);
a sensor (42) held by the support structure (44) under the lens (41), comprising a number of adjacent rows of pixels, where each row of pixels contains a number of pixels, and each pixel includes an image sensor; and
an image signal processor unit (46) directly connected to the sensing element (42) including an image scaling device configured to scale each row of pixels according to a scaling factor that differs from an adjacent row of pixels, so that the image is processed in the camera’s image pipeline. 2. The digital camera according to claim 1, in which:
the image scaling device is configured to scale each row of pixels according to a scaling factor having a value proportional to the position of the line between the start line and the end line. 3. The digital camera according to claim 1, in which:
the image scaling device is configured to be responsive to input of a scaling factor of the start line and a scaling factor of the end line, and comprises a calculation function configured to calculate scaling factors for each line between the start line and the end line. 4. The digital camera according to claim 1, in which:
an image scaling device is configured to calculate an input line length for a row of pixels as a ratio between a desired output line length common to all pixel lines and a scale factor for that line; and
Configurable to scale the image signals detected by the pixels of this line and which are within the input line length of the pixels to the desired output line length. 5. The digital camera according to claim 1, in which:
the image scaling device is configured to create an output image with centered lines. 6. The digital camera according to claim 5, in which:
the image scaling device is configured to calculate a central starting point for each input line from the sensor using the formula:

,
where start _n is the first pixel to be processed in line n; l is the number of pixels in the entire line; and l _n is the number of pixels to process in row n. 7. The digital camera according to claim 1, in which:
the camera module is formed by a support structure, and where the image signal processor is included in the camera module. 8. The digital camera according to claim 1, in which:
the image scaling device is configured to determine a position in a predetermined image format of the output pixel of a certain row of pixels, to determine the corresponding position in the detected image by back-scaling using the scaling factor for said certain row, and to determine the brightness value for the output pixel by interpolating the brightness values, which are detected by pixels adjacent to said corresponding position in detected image. 9. The digital camera according to claim 1, in which the image scaling device is configured to calculate the scaling factors, depending on a given estimated oblique angle between the image plane of the sensor and the object whose image you want to take. 10. The digital camera according to claim 1, in which the field of view of the camera is determined by the working area of the surface of the sensing element, which is offset from the center relative to the optical axis of the lens. 11. The digital camera according to claim 1, in which the image scaling device is configured to calculate the scaling factors S _n for each row n by the function S _n = m + n · k, where m and k are constants. 12. A communication device comprising:
housing;
a digital camera module according to any one of claims 1 to 11; and a central processor (20) configured to exchange information with said camera. 13. The communication device according to item 12, further comprising:
radio signal transceiver; and
a central processor configured to provide the transceiver of the radio signal with a scaled video signal from the digital camera module. 14. The communication device according to item 12, further comprising:
a display configured to display a scaled image that is provided by a digital camera module. 15. A method for capturing images using a digital camera of an electrical device, comprising the steps of:
direct the camera at the subject;
detecting image signals in a sensing element containing a number of adjacent rows of pixels, where each row of pixels contains a number of pixels, and each pixel includes an image sensor;
processing the detected image signals in the camera image pipeline by scaling each pixel timing according to a scaling factor that differs from the adjacent pixel row to provide a scaled image using an image signal processor connected directly to the sensor; and
output a scaled image using a central processor. 16. The method according to clause 15, further containing a stage in which:
scaling each row of pixels according to a scaling factor having a value proportional to the position of the row between the start row and the end row. 17. The method of claim 15, comprising the steps of:
set the scaling factor of the start line and the scaling factor of the end line; and
calculating scaling factors for each line between the start line and the end line. 18. The method of claim 15, comprising the steps of:
compute the input string length for the pixel string as
the ratio between the desired output line length, common to all lines of pixels, and the scaling factor for this line; and
scale the image signals detected by the pixels of this line, and which are within the input line length of the pixels, to the desired output line length. 19. The method according to clause 15, containing a stage in which:
provide a scaled image with centered lines. 20. The method according to clause 15, containing the stage at which:
calculate the center starting point for each input line from the sensor using the formula:

,
where start _n is the first pixel to be processed in line n; l is the number of pixels in the entire line; and l _n is the number of pixels to process in row n. 21. The method according to clause 15, containing a stage in which:
process the detected image by the image signal processor, made as a unit with a digital camera in the camera module of the communication device. 22. The method according to clause 15, containing the stage on which:
transmitting the scaled image to the remote receiver using the radio transceiver in the radio terminal. 23. The method according to clause 15, containing a stage in which:
represent a scaled image on the display. 24. The method of claim 15, comprising the steps of:
set the image format;
determine the position of the output pixel in a row of pixels in the image format;
determining an appropriate position in the detected image by backscaling using the scaling factor for said row; and
determining a brightness value for the output pixel by interpolating the brightness values that are detected by pixels adjacent to said corresponding position in the detected image. 25. The method according to clause 15, containing the stage on which:
scaling factors are calculated depending on a predetermined estimated oblique angle between the image plane of the sensor and the object whose image you want to take.

Images