KR20160100785A - Data Processing Device For Processing Multiple Sensor Data and System including the Same - Google Patents

Abstract

Disclosed are a data processing device for processing a plurality of sensor data, capable of efficiently processing multiple pieces of homogeneous or heterogeneous sensor data, and a data processing system including the same. The device comprises: first to k^th (k is an integer equal to or greater than two) sensors; a first and a second pre-processor receiving at least one piece of sensor data from the first to k^th sensors to pre-process the received sensor data; a first switching circuit connected between the first to k^th sensors, and the first and the second pre-processor, and selectively inputting at least one piece of sensor data received from the first to k^th sensors to the first and the second pre-processor in response to a control signal; and a hybrid data processing engine receiving and processing first and second pre-processed data pre-processed by the first and the second pre-processor together.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data processing apparatus for processing a plurality of sensor data and a data processing system including the apparatus,

An embodiment according to the concept of the present invention relates to a data processing apparatus for processing two or more sensor data and a system including the data processing apparatus.

As the digital camera technology advances, beyond the digitalization of the functions of the existing analog camera, the range is expanding to various new fields.

In particular, various sensors are emerging in order to overcome the limitations of CCD (charge coupled device) sensor and CIS (CMOS image sensor). Accordingly, smart phones and portable camera devices employing two or more sensors are also emerging.

SUMMARY OF THE INVENTION The present invention provides a data processing apparatus capable of efficiently processing a plurality of identical or different sensor data, and a system including the same.

Another object of the present invention is to provide a data processing apparatus and a system capable of processing data in a general manner corresponding to various sensor types and numbers, reducing memory access and reducing memory bandwidth and power.

According to an embodiment of the present invention, there is provided a data processing apparatus comprising: a plurality of preprocessors for performing correction processing on a plurality of sensor data; A first switching circuit for selectively mapping a plurality of sensor data output from at least two sensors to at least two preprocessors among the plurality of preprocessors; And a hybrid data processing engine that performs at least one of image enhancement and distance information determination on the plurality of sensor data received from the at least two preprocessors.

The first switching circuit may selectively map the plurality of sensor data output from at least two different types of sensors.

The first switching circuit may selectively map the plurality of sensor data to at least two preprocessors of the plurality of preprocessors according to the types of the plurality of sensor data.

When the first sensor data from the first sensor and the second sensor data from the second sensor are inputted to the first switching circuit, the first switching circuit outputs the first sensor data and the second sensor data to the And can be input to the first and second preprocessors, respectively.

When the second sensor data from the second sensor and the third sensor data from the third sensor are input to the first switching circuit, the first switching circuit outputs the second sensor data and the third sensor data to the first and second free Processor, respectively.

Each of the plurality of pre-processors corrects a difference in light and shade due to lens shading; A defective pixel correcting unit for correcting the pixel value of the defective pixel; And a chromatic aberration correcting unit for correcting chromatic aberration of the lens.

Wherein the first preprocessor of the plurality of preprocessors comprises a first processor for calibrating first sensor data input to the first preprocessor based on second sensor data input to a second preprocessor among the plurality of preprocessors, And a calibration engine.

The second preprocessor may not include a calibration engine.

The first calibration engine may calibrate the first sensor data by performing at least one of transition, rotation, and scaling on the first sensor data.

The first calibration engine may calibrate the first sensor data based on the second sensor data received from the second preprocessor to the first calibration engine.

Wherein the second preprocessor further comprises a second calibration engine wherein the first calibration engine calibrates the first sensor data based on a control signal received from a register or a CPU, 2 Sensor data can be calibrated.

The hybrid data processing engine includes a plurality of hardware unit processors for performing at least one of image enhancement and distance information determination.

The connection between the plurality of hardware unit processors and the operation of the plurality of hardware unit processors may vary depending on the operation mode of the data processing unit.

The data processing apparatus may further include a register for selectively controlling connection between the plurality of hardware unit processors and operation of the plurality of hardware unit processors according to the operation mode.

The data processing apparatus may further include a plurality of transducers for selectively converting at least one of a resolution and a pixel format of the plurality of sensor data received from the at least two preprocessors.

The data processing apparatus further includes a second switching circuit for selectively mapping the plurality of sensor data from the at least two preprocessors to the plurality of converters according to an operation mode of the data processing apparatus .

The hybrid data processing engine may receive the plurality of sensor data on-the-fly, without memory access from the at least two preprocessors.

The hybrid data processing engine may receive the plurality of sensor data stored in the memory from the at least two preprocessors through memory access. According to an embodiment of the present invention, A first and a second preprocessor each receiving and pre-processing sensor data of at least one of the first through k-th (k is an integer of 2 or more) sensors; (K is an integer greater than or equal to 2) sensors, and at least one of the first through k-th (k is an integer of 2 or more) sensors is connected between the first through k-th (k is an integer of 2 or more) sensors and the first and second pre- A first switching circuit for selectively inputting sensor data to the first and second pre-processors; And a hybrid data processing engine for receiving and processing the first and second preprocessed data preprocessed by the first and second preprocessor.

According to the embodiment, in the first mode, the first and second sensors of the first through k-th (k is an integer of 2 or more) sensors are selected, and in the second mode, One sensor is selected and the first switching circuit connects the sensor selected in the first mode or the second mode to the first and second preprocessors in response to the control signal.

According to an embodiment, at least one of the first and second preprocessors includes a calibration engine that calibrates the first or second sensor data to match the other sensor data. According to an embodiment, the first and second At least one of the preprocessors corrects a difference in light and shade caused by a difference in amount of light due to a curved surface of the lens; A defective pixel correcting unit for correcting pixel values of defective pixels and generating corrected data; And a chromatic aberration correcting unit for correcting chromatic aberration of the lens.

According to an embodiment, the hybrid data processing engine may perform noise reduction, de-focusing, dynamic range enhancement, contrast enhancement, and the like on the first and second pre- ) Of the image data.

According to an embodiment, the hybrid data processing engine includes a plurality of unit processors, and the connection relationships of the plurality of unit processors vary according to a mode, and perform different functions (or operations) according to the modes.

The hybrid data processing engine may further include at least one connection switching circuit for connecting the plurality of unit processors differently according to a connection control signal according to the mode.

The data processing apparatus may further include a converter connected between the first preprocessor or the second preprocessor and the hybrid data processing engine and for changing the resolution of the output data of the first preprocessor.

The data processing apparatus may further include a second switching circuit for selectively inputting the output data of the first and second preprocessors to the converter or the hybrid data processing engine in response to a second control signal have.

The sensor data output from the first to k-th (k is an integer of 2 or more) sensors may be input to the first and second preprocessors on-the-fly without going through a memory.

According to an embodiment, each of the first and second preprocessors performs geometric correction and / or optical correction on incoming sensor data.

According to the embodiment of the present invention, first to k-th (k is an integer of 2 or more) sensor: each sensor data of the selected one of the first to k-th (k is an integer of 2 or more) a first to an m-th preprocessor for performing a plurality of processes; A first switching circuit for selectively mapping at least two sensors among the first through k-th sensors and at least two preprocessors among the first through m-th preprocessors according to a first control signal; A hybrid data processing engine for outputting an improved image or extracting information using two or more preprocessed data preprocessed by the first through m-th preprocessors; And a processor for generating the first control signal in accordance with the mode.

According to an embodiment, at least one of the first to m-th pre-processors includes a calibration engine for calibrating input sensor data according to other sensor data.

According to an embodiment, the data paths output from the first through k-th (k is an integer greater than or equal to 2) sensors and input to the hybrid data processing engine are connected on-the-fly without going through a memory .

According to an embodiment, when the mode is a first mode, the hybrid data processing engine uses first and second preprocessed data preprocessed by the first and second preprocessors, The disparity map data can be output by obtaining the parity (disparity).

Wherein the first pre-processor is connected to the first sensor and the second preprocessor is connected to the second sensor when the mode is the second mode, and the hybrid data processing engine is connected to the first and second pre- The high dynamic range image is generated using the first and second preprocessed data pre-processed by the processor, and the exposure time of the first sensor and the exposure time of the second sensor may be set differently.

Each of the first and second preprocessors may perform geometric correction and / or optical correction on the input sensor data.

According to an embodiment of the present invention, a data processing system comprising a first through k-th (k is an integer greater than or equal to 2) sensor and a system-on-chip coupled to the first through k-th / RTI >

Wherein the system-on-chip comprises: first through m-th preprocessors each receiving and pre-processing sensor data of a selected one of the first through k-th (k is an integer of 2 or more) sensors; A first switching circuit for selectively mapping at least two sensors among the first through k-th sensors and at least two preprocessors among the first through m-th preprocessors according to a first control signal; A hybrid data processing engine for outputting an improved image or extracting information using two or more preprocessed data preprocessed by the first through m-th preprocessors; And a processor for determining one of the plurality of modes and for generating the first control signal according to the determined mode.

According to an embodiment, the processor determines the mode according to a driven application program or a selected menu.

According to an embodiment, the data processing system is a portable electronic device and the first through k (k is an integer greater than or equal to 2) sensors may comprise first and second sensors provided on the front side of the portable electronic device have.

The first to k-th (k is an integer of 2 or more) sensors may include third and fourth sensors provided on a rear portion of the portable electronic device.

If the determined mode is the first mode, the first preprocessor may be connected to the first sensor, and the second preprocessor may be connected to the second sensor.

If the mode is the second mode, the first preprocessor may be connected to the third sensor, and the second preprocessor may be connected to the fourth sensor.

According to the embodiment of the present invention, it is possible to provide a universal type that can correspond to various sensor types and numbers, and can reduce the system memory bandwidth and power consumption by minimizing memory input / output at the time of sensor input / output.

According to the embodiment of the present invention, a plurality of sensor data of the same or different type may be corrected by an arbitrary number of preprocessors (pre-processors), and then the corrected data may be combined to improve the image or generate additional information.

BRIEF DESCRIPTION OF THE DRAWINGS A brief description of each drawing is provided to more fully understand the drawings recited in the description of the invention.
FIG. 1A is a configuration block diagram of a data processing system according to an embodiment of the present invention. 1B is a modification of the data processing system according to an embodiment of the present invention.
FIG. 2 is a block diagram of a configuration of an embodiment of the first preprocessor shown in FIG. 1A.
FIG. 3A is a configuration block diagram showing an embodiment of the second preprocessor shown in FIG. 1A.
FIG. 3B is a block diagram showing another embodiment of the second preprocessor shown in FIG. 1A.
4 is a configuration block diagram showing an embodiment of the calibration engine shown in FIG.
5 and 6 are views for explaining the operation of the calibration engine shown in FIG.
Fig. 7 is a block diagram showing a configuration of an embodiment of the converter shown in Fig. 1A.
8A is a configuration block diagram illustrating an embodiment of the hybrid data processing engine shown in FIG.
8B is a block diagram showing another embodiment of the hybrid data processing engine shown in FIG.
9A is a configuration block diagram of a data processing system according to another embodiment of the present invention.
FIG. 9B is a modification of the data processing system shown in FIG. 9A.
10 is a configuration block diagram of a data processing system according to another embodiment of the present invention.
11 is a configuration block diagram showing an embodiment of the first preprocessor shown in FIG.
12 is a configuration block diagram showing an embodiment of the second preprocessor shown in FIG.
13 is a configuration block diagram of a data processing system according to another embodiment of the present invention.
14 is a diagram showing an example of the appearance of the data processing system shown in Fig.
15 is a conceptual diagram of an IoT service system in which a data processing apparatus according to an embodiment of the present invention can be used.
16 is a conceptual diagram of an IoT service system applicable to a vehicle according to an embodiment of the present invention.
17 is a conceptual diagram of a home network based IoT service system according to embodiments of the present invention.
18 is a schematic diagram for explaining a network between objects according to embodiments of the present invention.

It is to be understood that the specific structural or functional description of embodiments of the present invention disclosed herein is for illustrative purposes only and is not intended to limit the scope of the inventive concept But may be embodied in many different forms and is not limited to the embodiments set forth herein.

The embodiments according to the concept of the present invention can make various changes and can take various forms, so that the embodiments are illustrated in the drawings and described in detail herein. It should be understood, however, that it is not intended to limit the embodiments according to the concepts of the present invention to the particular forms disclosed, but includes all modifications, equivalents, or alternatives falling within the spirit and scope of the invention.

The terms first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms may be named for the purpose of distinguishing one element from another, for example, without departing from the scope of the right according to the concept of the present invention, the first element may be referred to as a second element, The component may also be referred to as a first component.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between. Other expressions that describe the relationship between components, such as "between" and "between" or "neighboring to" and "directly adjacent to" should be interpreted as well.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "having" and the like are used to specify that there are features, numbers, steps, operations, elements, parts or combinations thereof described herein, But do not preclude the presence or addition of one or more other features, integers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the meaning of the context in the relevant art and, unless explicitly defined herein, are to be interpreted as ideal or overly formal Do not.

1A is a configuration block diagram of a data processing system 10A according to an embodiment of the present invention. FIG. 1B is a variation 10A 'of a data processing system according to an embodiment of the present invention shown in FIG. 1A. FIG. 2 is a configuration block diagram showing an embodiment of the first preprocessor shown in FIG. 1A, and FIGS. 3A and 3B are configuration block diagrams showing an embodiment of the second preprocessor shown in FIG. 1A, 4 is a configuration block diagram showing an embodiment of the calibration engine shown in FIG. 5 and 6 are views for explaining the operation of the calibration engine shown in FIG. Fig. 7 is a block diagram showing a configuration of an embodiment of the converter shown in Fig. 1A. FIG. 8A is a configuration block diagram illustrating an embodiment 170A of the hybrid data processing engine shown in FIGS. 1A and 1B. FIG. 8B is a configuration block diagram illustrating another embodiment 170B of the hybrid data processing engine shown in FIGS. 1A and 1B.

Referring to FIGS. 1A, 1B, 2 A, 3 B, 4 through 7 and 8 A and 8 B, a data processing system 10 A, 10 A 'according to an embodiment of the present invention includes a plurality (k, (K is an integer equal to or greater than 2) and data processing apparatuses 100A and 100A '.

Each of the first through k-th sensors 111-1 through 111-k may be a homogeneous sensor (e.g., the same type of sensor) or a heterogeneous sensor (e.g., another type of sensor) .

Each of the first to k-th sensors 111-1 to 111-k includes a color image sensor for capturing a two-dimensional color image, a monochrome image sensor for capturing a two-dimensional monochrome image, an overfocus image sensor, a dynamic range (dynamic range) An infrared (IR) sensor for photographing an infrared region, or a depth sensor for measuring the distance of an image or an object. However, the present invention is not limited thereto, It is not. In addition, frame data, resolution, and exposure time of each of the first to k-th sensors 111-1 to 111-k may be different.

The dynamic range means the maximum contrast difference that can be expressed.

The autofocus image sensor may be a sensor having a phase detection function for improving autofocus.

The distance sensor is a sensor that measures a distance or a depth from an object. The distance sensor emits light (for example, infrared rays) as an object and receives light reflected by the object to receive a time of flight (TOF) But the present invention is not limited thereto.

The data processing apparatuses 100A and 100A 'can receive sensor data output from two or more sensors 110 and perform data processing such as preprocessing, size adjustment, and post-processing.

The data processing apparatus 100A includes a first switching circuit 120, first through m-th preprocessors 130-1 through 130-m, m is an integer of 2 or more, a second switching circuit 150, -1 to 160-n, where n is an integer greater than or equal to 1) and a hybrid data processing engine 170. In accordance with an embodiment of the present invention, additional components may be included, and one or more components (e.g., at least one of the second switching circuit 150 and transducers 160-1- through 160-n) may be omitted It is possible.

According to the first control signal CON1, the first switching circuit 120 receives at least two of the first through k-th sensors 111-1 through 111-k and the first through m-th preprocessors 130 -1 to 130-m) and selectively connects the at least two preprocessors. That is, the first switching circuit 120 outputs a plurality of sensor data output from at least two of the first through k-th sensors 111-1 through 111-k to the first through m-th preprocessors 130- 1 to 130-m).

For example, the first switching circuit 120 outputs the sensor data output from the two or more sensors selected from the first through k-th sensors 111-1 through 111-k to the first through m-th preprocessors 130-1 through 130- And the outputs (SO1 to SOk) of the first to k-th sensors and the inputs (PI1 to PIm) of the first to m-th preprocessors are input to the two or more preprocessors.

The first to third control signals CON1 to CON3 may be output from the CPU 190, but the present invention is not limited thereto. In the embodiment of FIG. 1B, the first to third control signals CON1 to CON3 may be generated by setting the register 195 in the data processing apparatus 100A '. The register 195 can be set by the CPU 190. [

K sensors 111-1 to 111-k according to the mode (for example, the operation mode of the data processing apparatus 100A or 100A 'or the data processing system 10A or 10A') The sensor can be different. In this case, the mode may be set or determined by the CPU 190, set in the register 195, or indicated via a control signal. Accordingly, the sensors connected to the first to m-th preprocessors 130-1 to 130-m by the first switching circuit 120 may also be changed.

According to an embodiment, m may be a value less than or equal to k.

For convenience of explanation, it is assumed that m is 2.

In one embodiment, only the first and second sensors 111-1 to 111-2 of the first to k-th sensors 111-1 to 111-k can be selected and operated. In this case, the first switching circuit 120 connects the output of the first sensor 111-1 to the input of the first preprocessor 130-1 in response to the first control signal CON1, The output of the sensor 111-2 may be connected to the input of the second preprocessor 130-2.

In the second mode, only the second and k-th sensors of the first to k-th sensors 111-1 to 111-k can be selected and operated. In this case, the first switching circuit 120 connects the output of the second sensor 111-2 to the input of the first preprocessor 130-1 in response to the first control signal CON1, The output of the sensor 111-k (e.g., the third sensor) may be connected to the input of the second preprocessor 130-2.

Each of the first through m-th preprocessors 130-1 through 130-m may perform at least one of geometrical correction and optical correction on input sensor data.

At least one of the first to m-th preprocessors 130-1 to 130-m can correct the input sensor data according to other sensor data. To this end, at least one of the first to m-th preprocessors 130-1 to 130-m may receive reference sensor information (REF_IF1 in FIG. 2 or REF_IF2 in FIG. 3B), which is information necessary for calibration.

For example, at least one of the first through m-th preprocessors 130-1 through 130-m receives reference sensor information (REF_IF1 in FIG. 2 or REF_IF2 in FIG. 3B) from the CPU 190 or the register 195, The input sensor data can be calibrated according to reference sensor information (REF_IF1 in FIG. 2 or REF_IF2 in FIG. 3B). According to the embodiment, each of the first to k-th sensors 111-1 to 111-k may store sensor characteristic information therein. The sensor characteristic information may include geometrical characteristics and optical characteristics of the sensor.

The CPU 190 reads the respective sensor characteristic information from the first to k-th sensors 111-1 to 111-k and outputs the sensor characteristic information to the first to m-th preprocessors 130-1 to 130- (REF_IF1 in Fig. 2 or REF_IF2 in Fig. 3B). Also, at least one of the first through m-th preprocessors 130-1 through 130-m receives sensor characteristic information or sensor data from other ones of the first through m-th preprocessors 130-1 through 130-m .

2, the first preprocessor 130-1 according to the embodiment of the present invention includes a lens shading corrector 131 and a bad pixel corrector 133 ), And a chromatic aberration corrector 135. The chromatic aberration corrector 135 may be a chromatic aberration corrector.

Lens shading is a phenomenon in which the lens becomes curved and darker due to insufficient amount of light going from the center to the periphery. The lens shading correcting unit 131 corrects the difference in lightness and darkness due to the difference in light quantity as described above.

The defective pixel correction unit 133 corrects dynamic defective pixels generated due to static defective pixels generated during the manufacture of the sensor or heat generation. The defective pixel correction unit 133 can detect a defective pixel and correct the pixel value of the detected defective pixel to generate a corrected pixel value.

The chromatic aberration correcting unit 135 corrects the chromatic aberration of the lens.

The chromatic aberration is aberration caused by the difference of the refractive index depending on the wavelength. The longer the wavelength of light is, the more the focus is focused away from the lens after passing through the lens. Therefore, the chromatic aberration correcting unit 135 corrects the chromatic aberration.

According to the embodiment, the first preprocessor 130-1 may further include a calibration engine 137. [

The calibration engine 137 performs an alignment function between the sensors.

The second preprocessor 130-2 according to the embodiment of the present invention includes a lens shading corrector 131 and a bad pixel corrector 133 ), And a chromatic aberration corrector 135. The chromatic aberration corrector 135 may be a chromatic aberration corrector.

The second preprocessor 130-2 shown in FIG. 3A differs from the first preprocessor 130-2 in that the calibration engine 137 is not included.

The lens shading correcting unit 131, the defective pixel correcting unit 133 and the chromatic aberration correcting unit 135 of the second preprocessor 130-2 are connected to the lens shading correcting unit 131 of the first preprocessor 130-1, The defective pixel correction unit 133, and the chromatic aberration correction unit 135, the description will focus on differences.

It is assumed that the first preprocessor 130-1 includes the calibration engine 137 and the second preprocessor 130-2 does not include the calibration engine 137. [

In the present embodiment, the calibration engine 137 of the first preprocessor 130-1 can correct the first sensor data SO1 to the second sensor data SO2. Here, "calibration" of the first sensor data SO1 may mean alignment such that the optical characteristics and the sensor characteristics of the first sensor 111-1 and the second sensor 111-2 become equal to each other have. Here, the second sensor data SO2 may be obtained by the second sensor 111-2, another pre-processor (e.g., the second preprocessor 130-2, the register 195, or the CPU 190) .

For example, when the foci of the first sensor 111-1 and the second sensor 111-2 are different, the calibration engine 137 sets the focal point of the first sensor data SO1 to the focal point of the second sensor data SO2 Can be corrected.

Alternatively, when the horizontal lines of the first sensor 111-1 and the second sensor 111-2 are different from each other, the calibration engine 137 sets the horizontal line of the first sensor data SO1 to the horizontal line of the second sensor data SO2 Can be corrected. To this end, the calibration engine 137 may receive reference sensor information REF_IF1 including reference horizontal line information from the second preprocessor 130-2. However, the embodiment of the present invention is not limited thereto, and the calibration engine 137 may receive the reference sensor information REF_IF1 from the second sensor 111-2, the register 195 or the CPU 190. [

In this manner, the calibration engine 137 corrects the input sensor data in accordance with the other sensor data, thereby making the output data between the heterogeneous sensors as output data of the homogeneous sensor, or the output data of the physical sensor such as the sensing position, time, The output data can be made so that the environment is the same as when each data is sensed under the same conditions. That is, it is possible to calibrate the sensed data from two or more sensors having at least one physical condition as well as the sensed data under the same conditions. In accordance with an embodiment, the calibration engine 137, as shown in FIG. 4, Two or more NxN matrix multipliers 138-1 through 138-p connected in a cascade form.

In the embodiment of FIG. 4, the calibration engine 137 includes first through pth (two or more integer) NxN matrix multipliers 138-1 through 138-p. The coefficients of the first to p < th > NxN matrix multipliers 138-1 to 138-p may be set by the CPU 190. [

According to the coefficients of the first to pth NxN matrix multipliers 138-1 to 138-p, the calibration engine 137 performs at least one of transition, rotation, and scaling The data RE1 can be calibrated.

5 and 6 show an example of data before and after calibration by the calibration engine 137, respectively. Referring to FIG. 5, FIG. 5A is the second sensor data SO2, and FIG. 5B is the first sensor data SO1 before calibration.

Before the calibration engine 137 corrects the first sensor data SO1, the horizontal lines of the first sensor data SO1 and the second sensor data SO2 are not aligned as shown in Fig. 5 .

Referring to FIG. 6, FIG. 6A is the second sensor data SO2, and FIG. 6B is the first sensor data SO1 after calibration.

The calibration engine 137 can correct the horizontal line of the first sensor data SO1 by rotating the first sensor data SO1 clockwise by a certain angle. The specific angle may be set according to the coefficients of the first to p < th > NxN matrix multipliers 138-1 to 138-p. After calibration by the calibration engine 137, the horizontal lines of the first sensor data SO1 and the second sensor data SO2 are aligned as shown in Fig.

According to the embodiment, unlike the second preprocessor 130-2 shown in FIG. 3A, the second preprocessor 130-2 'shown in FIG. 3B may also include a calibration engine 137-2 .

In this case, the first and second preprocessors 130-1 and 130-2 'can calibrate their respective sensor data according to their reference sensor information REF_IF1 and REF_IF2. The first and second reference sensor information REF_IF1 and REF_IF2 may be the same. In this case, the first and second preprocessors 130-1 and 130-2 'can calibrate data of different sensors under the same conditions.

Referring again to FIG. 1A, the second switching circuit 150 outputs the preprocessed data PO1 to POm output from the first to m-th preprocessors 130-1 to 130-m according to the second control signal CON2, The first to m-th preprocessors 130-1 to 130-m may be connected to one of the converters 160-1 to 160-n, where n is an integer of 1 or more, And the preprocessed data PO1 to POm output from the hybrid data processing engine 170 are directly connected to the hybrid data processing engine 170. [

That is, the second switching circuit 150 selectively connects the outputs (PO1 to POm) of the first to m-th preprocessors to the inputs (RI1 to RIn) of the converters 160-1 to 160- It is possible to bypass the converters 160-1 to 160-n.

Each of the first switching circuit 120 and the second switching circuit 150 may include a multiplier and / or a demultiplexer.

Each of the converters 160-1 to 160-n can convert the resolution and format of the input data RI1 to RIn.

Referring to FIG. 7, the converter 160-1 may include a size converter 161 and a format converter 163. The size converter 161 converts the spatial resolution of the input data RI1 according to the size information SIZE_IF and the format converter 163 converts the data RI1 input according to the format information FM_IF, Lt; / RTI > pixel format. The size information SIZE_IF and the format information FM_IF may be provided by the CPU 190 or the register 195. [

According to the embodiment, the connection relationship between the size converter 161 and the format converter 163 may be changed. For example, as shown in FIG. 7, the resolution may be adjusted first, then the format conversion may be performed, and the resolution may be adjusted after the format conversion is performed first.

According to another embodiment, size converter 161 or format converter 163 may be bypassed. For example, the size converter 161 may be bypassed, so that only the format conversion may be performed without resolution adjustment, and the format converter 163 may be bypassed so that only the resolution may be adjusted without format conversion.

According to the embodiment, each of the converters 160-1 to 160-n can be adjusted so that the spatial resolution between input data RI1 to RIn is the same. For example, when the resolution of the output data PO1 of the first preprocessor 111-1 is lower than that of the output data PO2 of the second preprocessor 111-2, the converter 160-1 The resolution of the output data PO1 of the first preprocessor 111-1 can be increased to be equal to the resolution of the output data PO2 of the second preprocessor 111-2.

As another example, when the resolution of the output data PO1 of the first preprocessor 111-1 is higher than that of the output data PO2 of the second preprocessor 111-2, the converter 160-1 The resolution of the output data PO1 of the first preprocessor 111-1 may be reduced to be the same as the resolution of the output data PO2 of the second preprocessor 111-2. Assuming that the resolutions of the first sensor 111-1 and the second sensor 111-2 are different, each of the converters 160-1 through 160-n may be up-scaled or down- the resolution of the first sensor 111-1 and the resolution of the second sensor 111-2 can be matched down-scaled.

According to the embodiment, each of the converters 160-1 to 160-n may equally match the pixel format of the input data RI1 to RIn. For example, the pixel format may be RGB444, ARGB888, YCbCr422, but is not limited thereto.

Hybrid data processing engine 170 receives two or more sensor data together to generate noise reduction, high dynamic range image generation, de-focusing, contrast extension, and the like Image depth enhancement, depth information extraction, and additional information such as object detection and recognition.

That is, the hybrid data processing engine 170 can output the enhanced image or extract information using two or more preprocessed sensor data preprocessed by the first through m-th preprocessors 130-1 through 130-m have.

For example, two or more pre-processed sensor data input to the hybrid data processing engine 170 may be processed by the converters 160-1 to 160-m in accordance with a control signal received from the register 195 or the CPU 190, May be adjusted data, or may be data by which the converters 160-1 to 160-m are bypassed. Depending on the embodiment, transducers 160-1 through 160-m may be omitted.

Referring to FIG. 8A, the hybrid data processing engine 170A according to an embodiment may include a plurality of (two or more) Processing Elements (PEs). Here, the plurality of unit processors (PEs) may be implemented in hardware.

The hybrid data processing engine 170A can perform more than two operations according to a combination of a plurality of unit processors.

Whether or not each of the plurality of unit processors is used and the connection relationship can be controlled by a CPU (190 in FIG. 1A) or a register (195 in FIG. 1B). For example, the CPU 190 may control whether or not each unit processor is used (or bypassed) and the connection of the unit processors according to the mode.

In the embodiment of FIG. 8A, the hybrid data processing engine 170A includes four unit processors, i.e., first through fourth unit processors 171, 172, 173, and 174, but the number of unit processors may be different.

In this embodiment, the hybrid data processing engine 170A includes third and fourth switch circuits 181 and 182 for controlling connection between the unit processors 171, 172, 173 and 174, The present invention is not limited thereto.

At least one of the first through fourth unit processors 171 through 174 may include at least one of noise reduction, de-focusing, dynamic range improvement, and contrast enhancement Can be performed.

According to the embodiment, in the first mode, the hybrid data processing engine 170A can output the image with the improved dynamic range using the first and second unit processors 171 and 172. [ In this case, the third to fourth unit processors 173 and 174 may be bypassed in response to the corresponding control signals CPEc and CPEd.

In the second mode, the hybrid data processing engine 170A can calculate disparity map data using the first, third, and fourth unit processors 171, 173, and 164.

As described above, according to the embodiment of the present invention, the hybrid data processing engine 170A can perform a different function by combining different unit processors, each having a specific image processing function, have.

Referring to FIG. 8B, the hybrid data processing engine 170B may be implemented as a field programmable gate array (FPGA) based on magnetic random access memory (MRAM). For example, the hybrid data processing engine 170B may include a gate array comprised of a plurality of logic gates G-11 through G-hg. The connection between each logic gate (G-11 through G-hg) is programmable. Therefore, depending on the mode, different operations can be performed by differentiating the connections between the logic gates (G-11 to G-hg) of the hybrid data processing engine 170B. Each of the logic gates G-11 to G-hg may be a circuit that performs a specific logic operation (e.g., a logical sum, a logical product, an exclusive-or).

According to an embodiment, the data path between each of the components of FIG. 1A may be connected on-the-fly without going through a memory (not shown).

For example, the first through the m-th preprocessors 13-1 through 13-m are connected to the first through m-th preprocessors 130-1 through 130-m in the first through k- m to converters 160-1 to 160-n and each data path from converters 160-1 to 160-n to hybrid data processing engine 170, 170A or 170B is on-the-fly -the-fly. In addition, the hybrid data processing engine 170, 170A or 170B may generate a plurality of sensor data from at least two of the preprocessors 130-1 through 130-m in an on-the-fly manner, Lt; / RTI >

Thus, when each data path is connected on-the-fly, there is no need to write and read data in the memory (e.g., the external memory of the data processing apparatus 100A or 100A '), the bandwidth is reduced, and the power consumption due to the memory access is reduced.

However, according to an embodiment, a DMA may be provided in at least one of the data paths so that data may be transferred from one component to another via memory.

9A is a configuration block diagram of a data processing system according to another embodiment of the present invention. 9A, the output data PO1 to POm of the first to m-th preprocessors 130-1 to 130-m in the data processing system 100B are stored in the memory 15, n converters 160-1 to 160-n can read and process the output data PO1 to POm of the first to m-th preprocessors 130-1 to 130-m from the memory 15. [

FIG. 9B is a modification of the data processing system according to the embodiment of the present invention shown in FIG. 9A. 9B, the output data RO1 to ROn of the first to nth converters 160-1 to 160-n in the data processing system 100B 'are stored in the memory 15, and the hybrid data processing engine 100B' The controller 170 may read the output data RO1 to ROn of the first to nth converters 160-1 to 160-n from the memory 15 and process the same.

As described above, data transmission between one component and another component in the data processing system 10A, 10A ', 10B may be performed in an on-the-fly manner, but may also be performed in the memory 15. Also, according to the embodiment, the connection between the plurality of sensors 111-1 to 111-k and the data processing apparatuses 100A, 100A ', 100B, and 100B' may be connected by wire and / or wirelessly.

In the embodiment of Figs. 1A, 1B, 9A or 9B, each component can be bypassed according to the control of the CPU 190 or the setting of the register 195.

In the embodiment of FIG. 1A, the CPU 190 is provided outside the data processing apparatus 100A, but may be included in the data processing apparatus 100A according to the embodiment.

10 is a configuration block diagram of a data processing system 20 according to another embodiment of the present invention. FIG. 11 is a configuration block diagram showing an embodiment of the first preprocessor shown in FIG. 10, and FIG. 12 is a configuration block diagram showing an embodiment of the second preprocessor shown in FIG.

10 to 12, a data processing system 20 according to an embodiment of the present invention includes first and second sensors 211 - 1 to 211 - 2 and a data processing apparatus 200.

The first sensor (Sensor R) 211-1 is a camera sensor corresponding to the right eye, the second sensor (Sensor L) 211-2 is a camera sensor corresponding to the left eye, and the first and second sensors The focal lengths 211-1 and 211-2 may have different focal lengths, optical characteristics of lenses, and distortion or signal characteristics of the sensor itself. The data processing apparatus 200 includes a first switching circuit 220, first and second preprocessors 230-1 through 230-2, a second switching circuit 250, a converter 260, and a hybrid data processing engine 270). According to an embodiment of the present invention, additional components may be included, and one or more components (e.g., at least one of second switching circuit 250 and converter 260) may be omitted.

The configuration and operation of the data processing apparatus 200 are similar to those of the data processing apparatuses 100A and 100A 'of FIGS. 1A and 1B, and therefore, differences will be mainly described.

10, 11 and 12, each of the first and second preprocessors 230-1 to 230-2 includes a lens shading correction unit 231, a defective pixel correction unit 233, (235) and a calibration engine (237-R or 237-L).

The configurations of the first and second preprocessors 230-1 to 230-2 are similar to those of the first and second preprocessors 130-1 to 130-2 shown in FIGS. Do.

However, in the first and second preprocessors 130-1 to 130-2 shown in FIGS. 2 and 3A, the calibration engine 137 is provided only in the first preprocessor 130-1, The first and second preprocessors 230-1 to 230-2 shown in FIG. 12 are each provided with a calibration engine 137-R or 137-L.

In this case, it is possible to align the first and second sensor data SRO and SLO by correcting the input image of the first calibration engine 237-R and the second calibration engine 237-L, respectively have.

For example, in the case of a mode in which a stereo image is created using the first and second sensor data SRO and SLO, the first calibration engine 137-R and the second calibration engine 137-L are It is possible to perform the deviation correction to remove the vertical parallax of the first and second sensor data SRO and SLO.

The output of the first preprocessor 230-1 may be input to the converter 260 by the second switching circuit 250. The output of the second preprocessor 230-2 can be input to the hybrid data processing engine 270 by the second switching circuit 250. The converter 260 converts the output of the first preprocessor 230-1 The resolution of the output data of the first preprocessor 230-1 is made equal to the resolution of the output data of the second preprocessor 230-2 by up-scaling or down-scaling .

The hybrid data processing engine 270 receives and processes the output data of the converter 260 and the output data of the second preprocessor 230-2 together.

According to an embodiment, the converter 260 may be bypassed. That is, output data of the first preprocessor 230-1 may also be directly input to the hybrid data processing engine 270. [

The configuration and operation of the hybrid data processing engine 270 may be similar to that of the hybrid data processing engine 170 of FIG.

According to the embodiment, in the first mode, the hybrid data processing engine 270 obtains the disparity between each pixel from the two input data RRO and PLO and outputs disparity map data or distance information can do.

According to the embodiment, in the second mode, the exposure times of the first and second sensors 211-1 and 211-2 are set differently, and the hybrid data processing engine 270 generates two input data RRO and PLO, A high dynamic range image can be generated. In this case, the exposure time of the first sensor 211-1 may be long and the exposure time of the second sensor 211-2 may be short.

According to the embodiment of the present invention described above, a plurality of sensor data of the same or different type may be corrected by an arbitrary number of preprocessors, and the corrected data may be combined to improve the image or generate additional information.

13 is a configuration block diagram of a data processing system according to another embodiment of the present invention. 14 is a diagram showing an example of the appearance of the data processing system shown in Fig.

13 and 14, the data processing system 30 includes a mobile terminal such as a smart phone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA) A mobile internet device (MID), an e-book, a portable multimedia player (PMP), or a digital camera.

The data processing system 30 may include an application processor 300, a plurality of sensors 110, a display device 370, and a memory 380.

As shown in FIG. 14, the plurality of sensors 110 are provided on the first and second camera sensors 111-1 and 111-2 and the rear portion 30B provided on the front portion 30A of the data processing system And may include third and fourth camera sensors 111-3 and 111-4.

The first and second camera sensors 111-1 and 111-2 provided on the front portion 30A recognize a face of a user or obtain a stereo image of a background or an object located in front of the data processing system 30 Can be used.

The third and fourth camera sensors 111-3 and 111-4 provided on the rear portion 30A are used to obtain a background image or a stereo image of an object located behind the data processing system 30, Can be used to obtain a high dynamic range image with different conditions such as exposure time and focus of the sensors 111-3 and 111-4.

Therefore, according to the application program 300 operated by the user in the data processing system 30 or the menu to be selected, the data processing system 30 selects two or more sensors among the plurality of sensors 111-1 through 111-3 And other functions can be realized by combining sensor data output from two or more sensors.

In order to implement the different operations, the first switching circuit 220 and the second switching circuit 250, as described above in FIGS. 1A, 1B, 2, 3A, 3B, 4-7 and 8A and 8B, The connection relationships of the data paths, and the use and connection relationships of the plurality of unit processors of the hybrid data processing engine 170 may be set differently.

The application processor 300 includes a central processing unit (CPU) 310, a read only memory 320, a random access memory 330, a data processing unit (DPD) 100, a sensor interface 340, A display interface 350, and a memory interface 360.

The application processor 300 may be implemented as a system on chip (SoC). Each of the configurations 310, 320, 330, 100A, 340, 350, and 360 of the application processor 300 can exchange data with each other via a bus 305. [

The CPU 310 can control the overall operation of the application processor 300. [ For example, the CPU 310 may process or execute programs and / or data stored in the ROM 320 and / or the RAM 330. [

According to an embodiment, CPU 310 may be implemented as a single computing component, i.e., a multi-core processor, having two or more independent processors (or cores).

ROM 320 may store programs and / or data that are continuously used. According to an embodiment, the ROM 320 may be implemented as an erasable programmable ROM (EPROM) or an electrically erasable programmable ROM (EEPROM).

RAM 330 may temporarily store programs, data, and / or instructions. According to an embodiment, the RAM 330 may be implemented as dynamic RAM (DRAM) or static RAM (SRAM).

The RAM 330 may be input / output through the interfaces 340, 350, and 360, or may temporarily store data generated by the CPU 310.

The data processing apparatus 100 refers to the data processing apparatuses 100A, 100A ', 100B, and 100B' shown in Figs. 1A, 1B, 9A, or 9B. The data processing apparatus 100 performs preprocessing, size conversion, and processing on the data received from the first to k-th sensors 111-1 to 111-k, and supplies the processed image data to the RAM 330, The interface 350, or the memory interface 360. FIG.

According to the embodiment, the data processing apparatus 100 may be the data processing apparatus 200 shown in Fig.

The sensor interface 340 may control the first to k-th sensors 111-1 to 111-k. According to the embodiment, the sensor interface 340 and the data processing apparatus 100 may be implemented as a single module.

The display interface 350 may interface data (e.g., image data) output to the display device 370 external to the application processor 300.

The display device 370 may output data for an image or an image through a display such as a liquid crystal display (LCD) or an active matrix organic light emitting diode (AMOLED).

The memory interface 360 may interface data input from memory 380 external to application processor 300 or data output to memory 380.

According to an embodiment, the memory 380 may be implemented as a non-volatile memory, such as a flash memory or a resistive memory.

As described above, the data processing system according to the embodiment of the present invention includes two or more sensors for capturing the user himself or herself on the front face and recognizing the face of the photographer. In order to obtain a stereo image or the like, A sensor may be additionally provided. According to the embodiment of the present invention, when a plurality of sensors are provided, various functions can be implemented by using different combinations of data processors, instead of having a data processor corresponding to each sensor on a one-to-one basis.

Therefore, since a separate data processing device corresponding to each sensor is not required, it is possible to prevent the implementation of redundant functions or operations, and thereby the increase in size and power consumption.

Further, according to embodiments of the present invention, memory accesses on the data path between the output of the sensor and the hybrid data processing engine can be minimized to prevent bandwidth increase and power consumption increase for memory input and output .

FIG. 15 is a conceptual diagram of an Internet of Thinns service system 500 in which a data processing apparatus 100A, 100A ', 100B, 100B' or 200 according to an embodiment of the present invention can be used. The embodiment of FIG. 15 shows usage scenarios for health, personal safety, social network service (SNS), information provision, and smart home service. 15, an IoT service system 500 may include at least one IoT device 510, a gateway 525, a server 540, and at least one service provider 550, 560, 570.

The IoT device 510 includes a smart glass 510-1, an earphone 510-2, an ECG / PPG 510-3, a waistband 510-4, a band or a clock 510-5 Such as a blood glucose meter 510-6, a thermostat 510-7, a shoe 510-8, a necklace 510-9, and the like. The wearable device 510 may include a sensor that senses the state of the user 520, the ambient environment, and / or the user command. The sensors included in each wearable device 510 may correspond to the first to k-th sensors 111-1 to 111-k. In addition, the IoT device 510 may include a replaceable battery for power supply, or may include a wireless charging function, and may include a wireless communication function for communicating with the outside.

The gateway 525 may transmit the collected information from the sensors to the server 540 via the communication network or may transmit the analysis information transmitted from the server 540 to the corresponding IoT device. For example, the gateway 525 may be coupled to the IoT device through a short-range wireless communication protocol. The gateway 525 may be a smartphone connectable to a wireless communication network such as Wi-fi, 3G, or LTE. The gateway 525 may include the data processing apparatus 100 according to an embodiment of the present invention. The data processing apparatus 100 of the gateway 525 performs correction / calibration and size conversion / resolution conversion of the sensor data received from the plurality of sensors, and post-processes and combines the corrected / corrected / can do.

The gateway 525 may be connected to the server 540 via an Internet network or a wireless communication network. The server 540 may store or analyze the collected information to generate associated service information or provide stored and / or analyzed information to the service provider 550, 560, 570. The service provider 550, 560, 570 may analyze the collected information and provide the service to the user 520. Here, the service may be provision of useful information for the user 520, provision of an alarm, provision of personal protection information, or provision of control information of the wearable IOT device 510.

The smart glass 510-1 is worn on the head of the user 520 and includes a dry eye sensor, an eye blink sensor, an image sensor, a brainwave sensor, Sensing the user's surroundings or the state of the user 520 and the commands of the user 520 via sensors such as a touch sensor, a voice recognition sensor and a global positioning system (GPS) can do. The sensed information may be transmitted to the server 540 and the server 540 may again provide the user 520 with a valid service. For example, the server 540 sends electrical stimulation information to the user 520 to remedy the abnormal brain waves based on the brain wave information of the received user 520, and transmits the electrical stimulation information to the user 520 through the smart glass 510-1. The user can adjust the mood of the user 520. [

The earphone 510-2 may be attached to the ear of the user 520 or may be attached in an ear-covering manner to a temperature sensor, an image sensor, and a touch sensor, a proximity sensor, a motion sensor, A gesture sensor, a heart rate sensor, and the like, to sense the body information and the commands of the user 520. [0050] The electrocardiogram meter (Electro Cardio Graphy (ECG) or Photo Plethysmo Gram (PPG) can measure the electrocardiogram of the user 520 using an electrocardiogram measuring sensor. The waistband 510-4 may include a waist circumference of the user 520, a breath measurement or an obesity measurement sensor, and may include a vibration function or an electric stimulation function for obesity or pain treatment. The band / clock 510-5 may include sensors associated with the user 520 temperature, heart rate, sleep, air pressure, ultraviolet radiation, oxygen saturation, optics, gyros, GSP, PPG, ECG, skin conduction, pedometer, And gas ejection for eradication. The blood glucose level tester 510-6 may include a sensor for measuring the glucose level of the user 520. The blood glucose measurement sensor may be a non-invasive sensor. The measured blood glucose may be transmitted to the server 540 through the smartphone / gateway 125 of the user 520.

The thermostat suit 510-7 may include a sensor that measures the temperature of the user 520 or the ambient temperature. The temperature regulating clothes 510-7 can control the cooling or heating function of the temperature regulating clothes 510-7 by comparing the preset temperature and the measured temperature. The thermostat cloth 510-7 may be, for example, an infant or adult diaper or undergarment. The diaper or undergarment may incorporate a skin conduction sensor, a temperature sensor, a test paper detection sensor, or a hydraulic pressure sensor to sense the state of the user 520 to inform the user of the replacement period of the diaper or underwear or perform cooling / heating . The diaper or undergarment may incorporate scorched heat and / or cooling pipes for cooling / heating.

The shoe 510-8 may include sensors such as the weight of the user 520, pressure at the sole region, air pollution in shoes, humidity, smell, GPS, steps, activity, The information collected at the sensor may be sent to the server 540 and the server 540 may send information to the user 520 such as an attitude correction for the user's posture or an alarm notifying the cleaning and replacement of shoes. The shoe may optionally provide the information to the user 520 via an application installed in the user smartphone / gateway 125.

The necklace 510-9 may be mounted on the neck of the user 520 and may include a sensor that senses the user's breath, pulse, body temperature, momentum, calories burned, GPS, EEG, voice, ECG, PPG, have. The information collected at the sensor may be analyzed by the IoT device itself or transmitted to the server 540 and the service provider 550, 560, 570 may send the associated service based on the user information received at the server 540 And provide it to the user 520. For example, the necklace 510-9 is attached to the dog and senses the voice of the dog, and the service provider can provide a voice translation service based on the sensed information. The translation service information may be reproduced through a speaker embedded in the necklace 510-9 or to an external audio device.

16 is a conceptual diagram of an IoT service system applicable to a vehicle according to an embodiment of the present invention. The embodiment of FIG. 16 shows usage scenarios for vehicle management, collision avoidance, vehicle service, and the like. Referring to Fig. 16, the service system 600 includes a vehicle 510 including a plurality of sensors 512-1 through 512-G. The plurality of sensors 512-1 to 512-G correspond to the plurality of sensors 111-1 to 111-k described above.

The service system 600 may also include an engine control unit 530, a server 540 and at least one service provider 560, 570.

The plurality of sensors 512-1 to 512-G are connected to at least one of the engine part sensor 512-1, the anti-collision sensors 512-4 to 512-11 and the vehicle travel sensors 512-12 to 512- One can be included. The plurality of sensors 512-1 to 512-G may further include a fuel level sensor 521-2 and / or an exhaust gas sensor 521-3.

The engine part sensor 512-1 may include an oxygen sensor, a coolant temperature sensor, a coolant level sensor, a manifold absolute pressure sensor (MAP sensor) (BARO pressure sensor (BPS), throttle position pressure sensor (TPS), mass air flow sensors (MAF), vane air flow sensor, (Eg, Karman Vortex Airflow Sensor), Knock Sensor, Air Temperature Sensor, Exhaust Gas Recirculation Valve Position Sensor (EGR), Crankshaft Position Sensor (CKP) And includes at least one of a camshaft position sensor, an engine oil level sensor, a mission oil level sensor, and a brake oil level sensor. Can.

The pressure sensor directly measures the atmospheric pressure and sends it to the ECU 530 to correct the fuel injection amount and the ignition timing. The MAP sensor provides the volume information to the ECU 530 using the pressure of the intake manifold, and the MAF sensor provides information on the mass of the intake air to the ECU 530 to determine the amount of fuel. The vane air flow sensor is a sensor in which the moving vane in the engine air intake system is connected to a variable resistor. The Kamman Vortex air flow sensor is an air flow sensor of hot wire type and / or hot wire type (hot film type). Knock Sensor is a kind of acceleration sensor that detects the occurrence of knocking in the engine. The exhaust gas recirculation valve position sensor sends a signal value from the oxygen sensor to the ECU 530 when CO or HC is high in the combustion gas and sends a signal value from the ECU 530 to the EGR solenoid valve to recirculate the exhaust gas . The crank position sensor (CKP) is a sensor that detects the engine speed and the exact position of the piston. The camshaft position sensor is a sensor for controlling fuel injection timing and ignition timing.

The anti-collision sensors 512-4 to 512-11 may include an airbag crash sensor, a front video camera, a back video camera, an infrared camera, a multi- A multi-beam laser, a long distance radar, a short distance radar, and an ultrasonic sensor.

The vehicle operation sensors 512-12 to 512-G include a GPS (Global Positioning System), a temperature sensor, a humidity sensor, a tire air pressure sensor, a steering angle sensor A wheel speed sensor (WSS or ABS), a vehicle speed sensor (VSS), a G sensor (G-force sensor), an electromechanical steering system, an electronic accelerator ), Electronic breaks, a pitch sensor, a height sensor (e.g., a wheel height sensor), an acceleration sensor, and a tilt sensor .

The engine control unit (ECU) 530 may collect the operation information 532 received from the plurality of sensors 512-1 to 512-G and transmit the collected operation information 532 to the server 540 via the communication network.

The server 540 may include the data processing apparatus 100 according to an embodiment of the present invention. The data processing apparatus 100 of the server 540 performs correction / calibration and size conversion / resolution conversion of the sensor data received from the plurality of sensors 512-1 to 512-G, The data can be post processed and combined to produce information. At this time, the engine control unit 530 and the server 540 can communicate the vehicle state information 534, the driver information 536, and / or the accident history information 538 with each other.

The service company 560 refers to the vehicle status information 534, the driver information 536 and / or the accident history information 538 stored in the server 540 to provide various services such as analyzed information and alarms to the corresponding vehicle can do. The service company 560 may share vehicle-related information stored in the server 540 with the contracted user 522. 17 is a conceptual diagram of a home network-based IoT service system 800 according to embodiments of the present invention.

Referring to FIG. 17, the IoT service system 800 includes a home network system 810 including a plurality of IoT devices 811 to 814. The IoT service system 800 may further include a communication network 850, a server 860, and a service provider 870.

The home network system 810 is a technology for controlling various devices in a building (a house, an apartment, a building, etc.) through a wired / wireless network and sharing content among devices. The home network system 810 may include a plurality of IoT devices 811 to 815, a home network 820, and a home gateway 830. In addition, the home network system 810 may further include a home server 840.

A plurality of IoT devices 811 to 814 may be used as home appliances 811 such as a refrigerator, a washing machine, an air conditioner, a stove, an oven, a dishwasher, a door lock, a CCTV, An entertainment device 813 such as a TV, an audio device, an audio / video device, a video device, a display device, a game device, a computer and the like, and a printer, a projector, a copying machine , Office equipment 814 such as a fax machine, a scanner, a multipurpose machine, and the like. In addition, the IoT devices 811 to 814 may include various electronic devices or sensing devices. In one embodiment, each of the IoT devices 811-814 may include at least one sensor. The sensors included in the IoT devices 811 to 814 correspond to the plurality of sensors 111-1 to 111-k in FIG. 1A.

The IoT devices 811 to 814 may communicate with each other through the home network 820 or may communicate with the home gateway 830. The IoT devices 811 to 814 and the home gateway 830 can transmit / receive sensor data or control information.

The home gateway 830 may include a data processing apparatus (100 of FIG. 1) according to an embodiment of the present invention. The data processing apparatus (100 in FIG. 1) of the home gateway 820 performs correction / calibration and size conversion / resolution conversion of the sensor data received from the IoT devices 811 to 814, converts the corrected / corrected / Processed and combined to generate information.

The home network 820 may include various wired / wireless communication networks. The IoT devices 811 to 814 may be connected to the external communication network 850 through the home gateway 830. The home gateway 830 can convert protocols between the home network 820 and the external communication network 840 and connect them to each other. In addition, the home gateway 830 may convert protocols between various communication networks included in the home network 820 to connect the IoT devices 811 to 814 and the home server 840. The home gateway 830 may be provided separately from other devices, or may be included in other devices. For example, the home gateway 830 may be included in the IoT devices 811 to 814 or the home server 840.

The home server 840 is provided within the home (or apartment complex) and can store or analyze the received data. The home server 840 may provide related services based on the analyzed information or may provide the analyzed information to the service provider 870 or the user equipment 880 via the external communication network 850. [ The home server 840 may store external content received through the home gateway 830, process the data, and provide the processed data to the IoT devices 811 to 814.

For example, the home server 840 may store input / output data provided from the security / safety device 812, or may be configured to perform automatic security services based on input / output data, power management services of the IoT devices 811 to 814, And so on. Also, the home server 840 analyzes the data provided from the IoT devices 811 to 814 including the illuminance, humidity, and pollution degree sensors, generates environment information, and provides the home environment control service based on the information Or provide the environment information to the user device 880. [

According to an embodiment, the home server 840 may include a data processing apparatus (100 of FIG. 1) in accordance with an embodiment of the present invention. The data processing apparatus 100 of the home server 840 performs correction / calibration and size conversion / resolution conversion of the sensor data received from the IoT devices 811 to 814, and performs post-processing of the corrected / corrected / And can combine to generate information.

The external communication network 850 may include the Internet and / or a public communication network. The public communication network may include a mobile cellular network. The external communication network 850 may be a channel through which information collected from the IoT devices 811 to 814 of the home network system 810 is transmitted.

The server 860 may store or analyze the collected information to generate associated service information or provide the stored information and / or analyzed information to the service provider 870 and / or the user device 880.

The service provider 870 may analyze the collected information and provide various services to the user. The service provider 870 may be a provider, a public facility, or the like that provides each service.

The service provider 870 provides services such as remote meter reading, crime prevention / disaster prevention, home care, healthcare, entertainment, education, and public administration to the IoT devices 811 to 814 through the home network system 810, To the user equipment (880). In addition, the service provider 870 may directly provide the service to the user.

18 is a schematic diagram for explaining a network between objects according to embodiments of the present invention.

Referring to FIG. 18, the network system 900 according to the present embodiment may include various IoT devices 910, 920, 930 and 940, and various wired / wireless communication technologies may be applied. Specifically, the Internet (IoT) technology including a sensor network, M2M communication, D2D communication, and the like may be applied to the network system 900.

The IoT devices 910, 920, 930 and 940 may be used for electronic devices such as a smart phone 910 and an air conditioner 940 as well as for devices equipped with sensors on conventional non-electronic devices, And bed / bedding 930 and the like. IoT devices 910, 920, 930, and 940 can communicate with each other to transmit / receive data.

The IoT devices 910, 920, 930, and 940 include at least one sensor, and can sense information generated inside / outside the IoT device through the sensor. The sensors mounted on the IOT devices 910, 920, 930, and 940 may be implemented as a smart sensor that includes not only the sensor elements but also a communication circuit or a processor. The sensors mounted on the IoT devices 910, 920, 930, and 940 correspond to the plurality of sensors 111-1 to 111-k in FIG. 1A.

In this embodiment, at least one of the IoT devices 910, 920, 930, and 940 may operate as a master device that controls the IoT devices 910, 920, 930, and 940. In one embodiment, the master device may be a smartphone 910, although the invention is not so limited. For example, in other embodiments, the master device may include smart home appliances such as tablet PCs, PDAs, notebooks and netbooks, mobile devices, wearable devices, or smart TVs. Hereinafter, the case where the master device is the smartphone 910 will be described in detail.

The smartphone 910 can generate a control signal or a notification signal based on sensed sensor data from various sensors installed therein or sensor data received from the IoT devices 920, 930, and 940 have. To this end, the smartphone 910 may include a data processing apparatus 100A, 100A ', 100B, 100B' or 200 according to an embodiment of the present invention.

The control signal may be a signal that controls the operation of the IoT devices 920, 930, 940 or the operation of the smartphone 910. The notification signal may be a signal indicating the status of the IoT devices 920, 930, 940 or the status of the user.

For example, the smartphone 910 receives sensing information such as temperature, humidity, user's breathing, heart rate, etc. from the bed / bedding 930 and displays the state of the user's sleeping state and surrounding environment on the basis of the sensing information It can be judged. The smartphone 910 may generate a control signal for controlling the operation of the air conditioner 940 and provide the control signal to the air conditioner 940 based on the determination result.

As another example, the smartphone 910 may generate a notification signal indicating the degree of contamination of the shoe 920 based on sensing information such as humidity, smell, pressure, position, and the like provided from the shoe 920, Calorie consumption, and the like can be generated.

The data processing apparatus 100A, 100A ', 100B, 100B' or 200 may be a smart device, a television, a mobile device, a computing device, an IoT device, a smart television, a wearable device, a scarfable device, , A portable multimedia player (PMP), an audio / video device, a set top box, a digital camera, an image capturing device, and the like, and any processing device.

The present invention can also be embodied as computer-readable codes on a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored.

Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like.

The computer readable recording medium may also be distributed over a networked computer system so that computer readable code can be stored and executed in a distributed manner.

Embodiments of the present invention may also be recorded in a computer program transmitted over a computer-readable transmission medium (e.g., a carrier wave) and received and implemented in a general purpose or dedicated digital computer executing the program have.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

10, 20, 30: data processing system
100, 200: data processing device
111-1 to 111-k: sensors
120, 150: switching circuit
130-1 to 130-m: preprocessor
160-1 to 160-n: converter
170: Hybrid data processing engine

Claims (20)

A plurality of preprocessors for performing correction processing on a plurality of sensor data;
A first switching circuit for selectively mapping a plurality of sensor data output from at least two sensors to at least two preprocessors among the plurality of preprocessors; And
And a hybrid data processing engine for performing at least one of image enhancement and distance information determination on the plurality of sensor data received from the at least two preprocessors. 2. The switching circuit according to claim 1, wherein the first switching circuit
And selectively mapping and inputting the plurality of sensor data output from at least two different types of sensors. 2. The switching circuit according to claim 1, wherein the first switching circuit
And selectively mapping the plurality of sensor data to at least two preprocessors of the plurality of preprocessors according to the types of the plurality of sensor data. 4. The apparatus of claim 3, wherein when the first sensor data from the first sensor and the second sensor data from the second sensor are input to the first switching circuit, the first switching circuit outputs the first sensor data and the second sensor data And maps the data to the first and second preprocessors, respectively, of the plurality of preprocessors. 5. The apparatus of claim 4, wherein when the second sensor data from the second sensor and the third sensor data from the third sensor are input to the first switching circuit, the first switching circuit outputs the second sensor data and the third sensor data And maps the data to the first and second preprocessors, respectively. 2. The apparatus of claim 1, wherein each of the plurality of preprocessors
A lens shading correcting unit for correcting a difference in light and shade due to lens shading;
A defective pixel correcting unit for correcting the pixel value of the defective pixel; And
And a chromatic aberration correcting unit for correcting chromatic aberration of the lens. 7. The method of claim 6, wherein the first of the plurality of preprocessors
And a first calibration engine for calibrating first sensor data input to the first preprocessor based on second sensor data input to a second preprocessor among the plurality of preprocessors. 8. The apparatus of claim 7, wherein the second preprocessor
A data processing apparatus that does not include a calibration engine. 8. The apparatus of claim 7, wherein the first calibration engine
Wherein the first sensor data is calibrated by performing at least one of transition, rotation, and scaling on the first sensor data. 8. The apparatus of claim 7, wherein the first calibration engine
And corrects the first sensor data based on the second sensor data received from the second preprocessor to the first calibration engine. 8. The apparatus of claim 7, wherein the second preprocessor
Further comprising a second calibration engine,
Wherein the first calibration engine calibrates the first sensor data and the second calibration engine calibrates the second sensor data based on a control signal received from a register or a CPU. 2. The method of claim 1, wherein the hybrid data processing engine
Wherein the plurality of hardware unit processors perform at least one of image enhancement and distance information determination. 13. The data processing apparatus according to claim 12, wherein the connection between the plurality of hardware unit processors and the operation of the plurality of hardware unit processors depend on an operation mode of the data processing apparatus. 14. The data processing apparatus according to claim 13, wherein the data processing apparatus
And a register for selectively controlling connection between the plurality of hardware unit processors and operation of the plurality of hardware unit processors according to the operation mode. The data processing apparatus according to claim 1, wherein the data processing apparatus
Further comprising a plurality of transducers for selectively converting at least one of a resolution and a pixel format of a plurality of sensor data received from the at least two preprocessors. 16. The data processing apparatus according to claim 15,
And a second switching circuit for selectively mapping the plurality of sensor data from the at least two preprocessors to the plurality of converters according to an operation mode of the data processing apparatus. 2. The method of claim 1, wherein the hybrid data processing engine
Wherein said plurality of sensor data is received on-the-fly, without memory access from said at least two preprocessors. 2. The method of claim 1, wherein the hybrid data processing engine
And receives the plurality of sensor data stored in the memory from the at least two preprocessors through memory access. First to kth (k is an integer of 2 or more) sensor:
First and second preprocessors each receiving and pre-processing sensor data of at least one of the first through k-th (k is an integer equal to or greater than 2) sensors;
(K is an integer greater than or equal to 2) sensors, and at least one of the first through k-th (k is an integer of 2 or more) sensors is connected between the first through k-th (k is an integer of 2 or more) sensors and the first and second pre- A first switching circuit for selectively inputting sensor data to the first and second pre-processors; And
And a hybrid data processing engine for receiving and processing first and second preprocessed data preprocessed by said first and second preprocessors. 20. The method of claim 19,
In the first mode, the first and second sensors of the first through k-th (k is an integer of 2 or more) sensors are selected, and in the second mode, at least one sensor different from the first and second sensors is selected And,
The first switching circuit
And in response to the control signal, connect a sensor selected in the first mode or the second mode to the first and second preprocessors.

Images