KR20200072845A - Mobile terminal and driving method thereof - Google Patents

Abstract

The present invention relates to a portable terminal and a driving method thereof. In particular, provided are the portable terminal capable of integrating an illuminance sensor of the portable terminal into a front camera by presenting a method of efficiently measuring illumination information from the front camera mounted on a front surface of a portable terminal and the driving method thereof.

Description

Mobile terminal and driving method thereof

The present invention relates to a portable terminal and a method of driving the same, and more specifically, by providing a method for efficiently measuring illumination information from a front camera mounted on the front of the portable terminal, the illumination sensor of the portable terminal can be integrated into the front camera. It relates to a portable terminal and a method for driving the same.

Thanks to advances in semiconductor technology and display technology, portable terminals have recently evolved into full screen displays.

Meanwhile, the front camera, proximity sensor, and illuminance sensor mounted on the front of the portable terminal cannot be placed in the display active area, which is an opaque part because it is an optical method. As a result, there is a Bezel area, which is a space for the arrangement of the front camera, proximity sensor, and illuminance sensor, which is an optical device, which is a obstacle to realizing a full screen display.

As a method for overcoming this, there is a method of constructing a hole for securing transmittance in the display active area, which is an opaque part, and reducing the bezel area by arranging optical devices disposed in the bezel in the hole area.

However, three or more holes are required to arrange the front camera, proximity sensor, and illuminance sensor, or there is a problem in that the hole size becomes large.

The above-mentioned background technology is technical information acquired by the inventor for the derivation of the present invention or acquired in the derivation process of the present invention, and is not necessarily a known technology disclosed to the general public before filing the present invention.

The present invention has been devised to solve the above-mentioned problems, and by presenting a method of integrating an illuminance sensor function in a front camera mounted on a portable terminal, to realize a full screen display of the portable terminal by minimizing the number of holes or the size of the hole. There is a purpose.

As a means for solving the above-described problems, the present invention has an embodiment with the following features.

The present invention, the front camera for photographing a subject in the front direction of the portable terminal; An image signal processing unit for processing the digital sensing image signal output from the front camera; And a central processing unit for determining an operation mode of the front camera as an illumination measurement mode or a camera mode, and receiving an image signal from the image signal processing unit to extract illumination information in the illumination measurement mode; It characterized in that it comprises.

The front camera includes an image sensor that converts the received light into an electrical signal, and in the illuminance measurement mode, the central processing unit selects and drives a part of the image sensor.

In the illuminance measurement mode, the partial area of the image sensor selected and driven by the central processing unit is characterized in that it consists of only G (green) pixels that sense green light.

Also, in the illumination measurement mode, the partial region of the image sensor selected and driven by the central processing unit is configured as a part of G pixels.

In addition, the portable terminal includes a tilt measurement sensor that detects tilt information of the portable terminal, and the central processing unit utilizes tilt information of the portable terminal received from the tilt measurement sensor to concentrate light on the image sensor. Characterized in that to drive by selecting.

Also, in the illuminance measurement mode, the central processing unit may simultaneously drive pixels selected as the partial region of the image sensor.

Also, in the illuminance measurement mode, the central processing unit sequentially drives pixels selected as the partial region of the image sensor.

In addition, the present invention, in a driving method of a portable terminal in which the front camera photographing an object in the front direction of the portable terminal operates in the illumination measurement mode or the camera mode, when the front camera is operated in the illumination measurement mode, the illuminance Setting a front camera of the portable terminal as a measurement mode operation condition; A detection image signal generation step of generating a digital detection image signal by taking a picture with a front camera under the illumination measurement mode operation condition; And an illuminance information extraction step of extracting information about illuminance from the digital sensing image signal. It characterized in that it comprises.

In the sensing image signal generating step, when driving an image sensor in which a plurality of pixel groups are arranged in a two-dimensional matrix form, a part of the image sensor is selected and driven.

The detecting image signal generating step includes a tilt measuring step of detecting tilt information of the portable terminal; And a selection driving step of selecting and driving an area where light is concentrated on the image sensor by using the tilt information of the portable terminal. It characterized in that it comprises.

The sensing image signal generating step is characterized in that the pixels selected as the partial regions of the image sensor are driven simultaneously or sequentially.

The present invention has an effect of minimizing the number of holes or the size of holes by presenting a method of integrating the illuminance sensor function in a front camera mounted on a portable terminal.

In addition, according to the present invention, by selecting and driving a partial area of the image sensor, the image sensor has an effect of shortening the time for generating a digital sensing image signal and signal processing time and reducing power consumption.

In addition, the present invention has an effect of increasing the accuracy of the illuminance measurement by selecting and driving an area in which light is concentrated in the image sensor by using the inclined angle information of the portable terminal.

In addition, the present invention has an effect of shortening the time for the image sensor to generate the digital sensing image signal and the signal processing time by simultaneously driving selected pixels as a partial region of the image sensor.

1 is a block diagram of a portable terminal according to an embodiment of the present invention
FIG. 2 is a diagram for driving all pixels in a pixel sensor array area in a camera mode according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating that some pixels of a pixel sensor array region are driven in an illuminance measurement mode according to an embodiment of the present invention.
FIG. 4 is another embodiment of FIG. 3, in which some pixels in the pixel sensor array area are driven by only G pixels.
5A to 5C are other embodiments of FIG. 4, which are sequentially driving pixel selection areas;
6 is a block diagram of a portable terminal according to another embodiment of the present invention
FIG. 7 is a view of vertically incident light on the front camera.
FIG. 8 is a diagram illustrating that a central portion of a pixel sensor array area is driven when the tilt of the portable terminal is in the state of FIG. 7.
9 is a view related to the front camera is tilted to the left and light enters at an inclined angle.
FIG. 10 is a view showing that a pixel in the left portion of the pixel sensor array area is driven when the inclination of the portable terminal is in the state of FIG. 9.
11 is a view related to the front camera is tilted to the right and the light is incident at an inclined angle.
FIG. 12 is a view showing that pixels in the right portion of the pixel sensor array area are driven when the inclination of the portable terminal is in the state of FIG. 11;
13 is a flowchart illustrating a method of driving a portable terminal according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, specific details for the practice of the present invention will be described.

Portable terminal

1 is a block diagram of a portable terminal according to an embodiment of the present invention.

The front camera of the present invention can selectively operate in the camera mode or the illumination sensor mode under the control of the central processing unit.

In the camera mode, it is the same as driving the front camera in a conventional portable terminal.

On the other hand, in the illumination sensor mode, the image sensor of the front camera is characterized in driving only pixels of a sub-group consisting of some pixels. Then, illuminance information is extracted from the digital sensing image signal generated by the pixels of the sub-group. This is because measuring the illuminance information does not require a digital sensing image signal from all pixels constituting the image sensor, so it is efficient in terms of computation time and power consumption to measure illuminance information by driving only some pixels.

The portable terminal includes a front camera 10, an image signal processor 20, and a central processor 30.

The front camera 10 is a device for photographing a subject in the front direction of a portable terminal, and converts the subject into an electric video signal. The front camera 10 is a lens (not shown) that collects or diverts light from a subject and transmits an image of the subject to the image sensor, blocks infrared components included in the optical signal passing through the lens, and generates the image from the image sensor. It includes an IR filter (not shown) that removes noise, and an image sensor 100 that converts light into electrical signals.

The image sensor 100 may be a CCD sensor or a CMOS sensor, but is preferably configured as a CMOS sensor capable of driving control by selecting only some pixels. This is because the present invention is characterized by efficiently measuring illuminance information by selecting and driving some pixels rather than driving all image sensors constituting the front camera in the illuminance sensor mode. Hereinafter, description will be given based on the CMOS sensor.

The image sensor 100 includes a pixel sensor array area 110 in which unit pixels including a light-receiving element are arranged in a two-dimensional matrix, a vertical driving unit 120 and a horizontal driving unit 130 for driving and controlling the array area, the It includes a control unit 140 for controlling the horizontal driving unit and the vertical driving unit.

The unit pixel includes a light-receiving element and a current-voltage conversion transistor that generates a voltage according to a photocurrent flowing through the light-receiving element to the drain. The light receiving element can absorb light and accumulate charge, and the accumulated charge is output to the outside through a transistor. The light receiving element may be a photodiode, a photogate, or a phototransistor. The unit pixels are arranged in a two-dimensional matrix as many as the number of pixels. For convenience of description, the pixels of the image sensor 100 are illustrated as 12 x 8. On the unit pixel, R, G, and B color filters are arranged with a specific pattern. In the RGB pattern, the R, G, R, and G patterns are repeated in one row, and the G, B, G, and B patterns are repeated in another row.

The vertical driving unit 120 controls the unit pixels of the pixel sensor array area 110 row by row by providing a driving signal for each row.

The horizontal driving unit 130 controls the unit pixels constituting the pixel sensor array region 110 by column by providing a driving signal for each column.

The control unit 140 may control the vertical driving unit 120 and the horizontal driving unit 130 to control driving of unit pixels constituting the pixel sensor array region 110 as a result.

The image signal processing unit 20 may include an ADC that converts an analog signal to a digital signal, and performs image processing on RAW data of the digital sensing image signal output from the image sensor 100. Image processing includes converting RGB data to YCbCr data, color correction, and the like.

The central processing unit 30 is in charge of overall control of the portable terminal. Specifically, the driving of each unit pixel of the image sensor 100 is controlled through the control unit 140, and the image signal processed by the image signal processing unit 20 is received to extract illumination information. In addition, the tilt information of the portable terminal is received from the tilt measurement sensor 50 to be described later, and the area of the unit pixel driven by the image sensor 100 is selected using the tilt information of the portable terminal.

FIG. 2 is a diagram for driving all pixels in a pixel sensor array area in a camera mode according to an embodiment of the present invention.

In the camera mode, the operation is the same as that of the front camera in a conventional portable terminal. As illustrated in FIG. 2, when the front camera operates in the camera mode, all pixels of the pixel sensor array area 110 are driven, and the image sensor 100 generates an image signal of a subject. In FIG. 2, an arrow displayed on a line connecting the vertical driving unit 120 and the horizontal driving unit 130 and the pixel sensor array area 110 represents that a driving signal is applied to a row or column of a corresponding pixel. The same applies to the drawings below.

3 is a diagram illustrating that some pixels of a pixel sensor array area are driven in an illuminance measurement mode according to an embodiment of the present invention.

When the front camera is operated in the illuminance measurement mode, not all pixels of the pixel sensor array area 110 constituting the image sensor are driven, but only some pixels of the selected area are driven by selecting some areas. The embodiment of FIG. 3 shows that the vertical driving unit 120 drives 2, 3, and 4 rows, and the horizontal driving unit 130 drives 2, 5, 8, and 10 columns. If the unit pixels to be driven are expressed in coordinates, in the order of rows and columns, (2,2), (2,5), (2,8), (2,10), (3,2), (3,5 ), (3,8), (3,10), (4,2), (4,5), (4,8), (4,10), and a total of 12 unit pixels are driven.

In measuring the illuminance, it is inefficient to extract the digital sensing image signal from all pixels and measure the illuminance, so it is sufficient to extract the illuminance information by driving only some pixels. The RAW data of the digitally sensed image signals output from some of the driven pixels, Data1 to Data4, are transmitted to the image signal processor 20, and the image-processed signals are transmitted to the central processor 30, so that illuminance information is extracted.

4 is a diagram of another embodiment of FIG. 3, in which some pixels of the pixel sensor array area 110 are driven by only G pixels.

The embodiment of FIG. 4 shows that the vertical drive unit 120 drives 2, 4, and 6 rows, and the horizontal drive unit 130 drives 3, 5, 9, and 11 columns. When the driven unit pixels are expressed in coordinates, in the order of rows and columns, (2,3), (2,5), (2,9), (2,11), (4,2), (4,5 ), (4,9), (4,11), (6,2), (6,5), (6,9), (6,11). The number of unit pixels to be driven is 12, which is the same as in the case of FIG. 3, but the difference is that all of the driven pixels are G pixels that detect green light.

The human eye is most sensitive to green light, and the illuminance sensor also uses a wavelength range of 525nm to 560nm, which corresponds to green light. Therefore, selecting and driving only some G pixels that detect green light is effective in extracting illumination information. That is, the embodiment of FIG. 4 has the same number of pixels as the embodiment of FIG. 3, but the illumination information can be extracted with higher accuracy. Alternatively, even if a smaller number of pixels are driven, the illuminance information can be extracted with the same accuracy as the embodiment of FIG. 3, thereby reducing power consumption.

In addition, under the control of the control unit 140, the vertical driving unit 120 is selected pixel regions, (2,3), (2,5), (2,9), (2,11), (4,3), Lines 2, 4, corresponding to (4,5), (4,9), (4,11), (6,3), (6,5), (6,9), (6,11), Six rows can be driven simultaneously or sequentially.

4 shows that the 2nd, 4th, and 6th rows were simultaneously driven. Here, the simultaneous driving means that the selected pixels are simultaneously operated through the control signals of the vertical driving unit 120 and the horizontal driving unit 130, and raw data generated from the selected pixels are simultaneously extracted in units of columns. The extracted raw data may be a value obtained by summing the information of each pixel unit for each column. Since pixel-level information is not required to extract the illumination information, driving at the same time does not interfere with extracting the illumination information. In this way, the effect obtained by simultaneously driving the 2nd, 4th, and 6th rows of the vertical driving unit 120 can shorten the time for the image signal processing unit 20 to extract raw data from the image sensor 100, and The image signal processing unit 20 has an advantage of reducing the time required to process raw data and reducing power consumption for processing the image.

5A to 5C are other embodiments of FIG. 4, and are diagrams for sequentially driving a pixel selection area.

5A shows that 2 rows, 5B 4 rows, and 5C 6 rows are driven. When the vertical driving unit 120 sequentially drives 2 rows, 4 rows, and 6 rows, raw data classified for each unit pixel to be driven may be obtained. 4, data corresponding to 3 columns was obtained as one raw data of G Data1. On the other hand, in the sequential driving embodiment of FIGS. 5A to 5C, data corresponding to 3 columns can be obtained as raw data of G Data11, G Data12, and G data13, which are raw data classified for each pixel (5, 9). , In the case of column 11). In this case, there may be a disadvantage in that the time for extracting the raw data and the time for processing the image increase, but there is an advantage in that the accuracy of the extracted data is high for measuring illuminance information.

6 is a block diagram of a portable terminal according to another embodiment of the present invention.

The tilt measurement sensor 50 detects tilt information of the portable terminal. The tilt measurement sensor may be formed of any one or a combination of a gyroscope sensor, a tilt measurement sensor, an acceleration sensor, a geomagnetic sensor, and a gravity sensor.

The image sensor 100, the image signal processor 20, and the central processor 30 are as described above in FIG.

FIG. 7 is a diagram for vertically incident light to the front camera, and FIG. 8 is a diagram for driving a central pixel of the pixel sensor array area when the inclination of the portable terminal is in the state of FIG. 7.

As described above, portable terminals have recently evolved into full screen displays. To this end, a hole H for securing transmittance is formed in the opaque display active area 11, and the front camera is disposed in the hole H area. Due to the structure of the hole H, the region where the light L incident on the image sensor 110 is concentrated varies depending on the inclination angle of the portable terminal.

In the case of the embodiment of FIG. 7, since light is vertically incident on the front camera, light is concentrated in the central portion of the sensor 110 relatively. Therefore, as shown in FIG. 8, it is preferable that the driven pixels are driven by selecting a central portion of the pixel array area 110. The tilt information of the portable terminal is measured from the tilt measurement sensor 50, and the measured tilt information is transmitted to the central processing unit 30. The central processing unit 30 selects the central portion of the pixel array region 110, which is an area where light is concentrated, by using the tilt information of the portable terminal, and instructs the control unit 140 to drive the selected region.

FIG. 9 is a view of the front camera tilting to the left and incident light at an inclined angle, and FIG. 10 is when the tilt of the portable terminal is in the state of FIG. 9, the pixel in the left part of the pixel sensor array area is driven It is a drawing about things.

In the case of the embodiment of FIG. 9, since light is incident on the front camera at an angle inclined to the left, the light is already concentrated on the left portion of the sensor 110 relatively. Therefore, as shown in FIG. 10, it is preferable that the driven pixel is driven by selecting the left portion of the pixel array area 110. The method of selecting and driving the left part is as described in the embodiments of FIGS. 7 to 8.

FIG. 11 is a diagram illustrating that the front camera is tilted to the right and incident light is tilted, and FIG. 12 is when the tilt of the portable terminal is in the state of FIG. 11, the pixel in the right portion of the pixel sensor array area is driven It is a drawing about things.

In the case of the embodiment of FIG. 11, since light is incident on the front camera at an angle inclined to the right, light is concentrated on the right portion of the sensor 110 relatively. Therefore, as shown in FIG. 12, it is preferable to select and drive the right part of the pixel array area 110. The method of selecting and driving the right part is as described in the embodiments of FIGS. 7 to 8.

How to drive a portable terminal

13 is a flowchart of a method of driving a portable terminal according to an embodiment of the present invention.

As shown in Fig. 13, the front camera of the portable terminal of the present invention selectively operates in the illumination measurement mode or the camera mode under the control of the central processing unit.

Here, the illuminance measurement mode is set as the default, so that the central processing unit operates the front camera in the camera mode only when the user commands the camera mode, and can be set to operate in the illuminance measurement mode otherwise.

When the front camera is determined to operate in the illuminance measurement mode and operates in the illuminance measurement mode, the driving method of the portable terminal includes a setting step (S210), a sensing image signal generation step (S220), and image processing of the sensing image signal. (S230), the illumination information extraction step of extracting information about the illumination from the image-processed digital sensing image signal (S240); It includes.

The setting step S210 is a step of setting the front camera mounted on the front of the portable terminal as an operating condition for illuminance measurement. For example, in order to measure ambient light consistently under the same conditions, the front camera can be set to be suitable for illuminance measurement, such as keeping the shooting conditions such as aperture, shutter, focus, and shooting cycle of the front camera constant at all times. have.

The sensing image signal generating step (S220) is a step of generating a digital sensing image signal by photographing a subject with a front camera under an illumination measurement mode operating condition.

As described above, the feature of the present invention is that the front camera generates a digital sensing image signal in the illumination measurement mode, and selects and drives a part of the image sensor 100. In addition, some areas of the image sensor 100 are characterized by consisting of only G (green) pixels that detect green light. In addition, some areas of the image sensor 100 are characterized by selecting an area in which light is concentrated on the image sensor 100 by utilizing the inclined angle information of the portable terminal. Further, in driving a selected pixel as a partial region of the image sensor 100, it is characterized in that it is driven simultaneously or sequentially.

The sensing image signal processing step (S230) is a step in which the image signal processing unit 20 performs image processing on RAW data of the digital sensing image signal output from the image sensor 100.

The illuminance information extraction step (S240) is a step in which the central processing unit 30 receives an image processed image signal from the image signal processing unit 20 and extracts illuminance information.

When the front camera is determined to operate in the camera mode, the method of driving the portable terminal when operating in the camera mode is the same as the method of driving the front camera by a conventional portable terminal.

It should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be interpreted as being included in the claims of the present invention. .

10: front camera
100: image sensor
110: pixel sensor array area
120: vertical drive
130: horizontal driving unit
140: control unit
20: image signal processing unit
30: central processing unit
50: tilt measurement sensor

Claims (14)

A front camera for photographing a subject in the front direction of the portable terminal;
An image signal processing unit for processing the digital sensing image signal output from the front camera; And
A central processing unit which determines an operation mode of the front camera as an illumination measurement mode or a camera mode, and extracts illumination information by receiving an image signal from the image signal processing unit in the illumination measurement mode; Portable terminal comprising a. According to claim 1,
The front camera includes an image sensor that converts the received light into an electrical signal,
In the illuminance measurement mode, the central processing unit selects and drives a partial area of the image sensor. According to claim 2,
A portable terminal, characterized in that the partial region of the image sensor selected and driven by the central processing unit in the illumination measurement mode consists of only G (green) pixels that sense green light. According to claim 3,
A portable terminal characterized in that the partial area of the image sensor selected and driven by the central processing unit in the illumination measurement mode is configured as a part of a G pixel. According to claim 1,
The portable terminal includes a tilt measurement sensor that detects tilt information of the portable terminal,
The central processing unit utilizes the tilt information of the portable terminal received from the tilt measurement sensor to select and drive an area where light is concentrated on the image sensor. According to claim 2,
In the illuminance measurement mode, the central processing unit simultaneously drives a pixel selected as the partial region of the image sensor. According to claim 2,
In the illumination measurement mode, the central processing unit sequentially drives a pixel selected as the partial region of the image sensor. In the driving method of a portable terminal in which the front camera photographing a subject in the front direction of the portable terminal operates in an illumination measurement mode or a camera mode,
When the front camera is operated in the illumination measurement mode,
Setting the front camera of the portable terminal as the operating condition of the illuminance measurement mode;
A detection image signal generation step of generating a digital detection image signal by taking a picture with a front camera under the illumination measurement mode operation condition; And
An illuminance information extraction step of extracting information about illuminance from the digital sensing image signal; Method of driving a portable terminal comprising a. The method of claim 8,
The sensing image signal generating method is a method of driving a portable terminal, wherein a plurality of pixel groups are driven by selecting a portion of an image sensor to drive an image sensor arranged in a two-dimensional matrix. The method of claim 9,
A method of driving a portable terminal, characterized in that the partial area of the image sensor is made of only G (green) pixels that sense green light. The method of claim 10,
A method of driving a portable terminal, characterized in that the partial region of the image sensor is configured as a part of a G pixel. The method of claim 8,
The detection image signal generation step
A tilt measurement step of detecting tilt information of the portable terminal; And
A selection driving step of selecting and driving an area where light is concentrated on the image sensor by using the tilt information of the portable terminal; Method of driving a portable terminal comprising a. The method of claim 9,
The sensing image signal generating step is a method of driving a portable terminal, characterized in that driving the selected pixel to the partial region of the image sensor at the same time. The method of claim 9,
The sensing image signal generating step sequentially drives a selected pixel to the partial region of the image sensor.

Images