JP2007201681A - Imaging device and method, recording medium and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To simultaneously acquire a picture and a light signal without deteriorating the resolution of the picture. <P>SOLUTION: A portable telephone 71 has a camera 81 arranged with approximately same angle of view at a place where an image sensor and an optical-communication sensor are close each other, a monitor 82, and an operation input 83. In the portable telephone 71, a picture signal obtained from the image sensor and communication information from a sending light source 3-1 or 3-2 received by the optical communication sensor are acquired by imaging a subject 2-1 and the subject 2-2 having the sending light sources 3-1 and 3-2 such as an LED respectively by the camera 81. In the portable telephone 71, the pictures 92-1 and 92-2 based on the communication information are superimposed on the picture 91 corresponding to the acquired picture signal, and displayed on the monitor 82. The method can be applied to the portable telephone with the camera section imaging the subjects. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an imaging apparatus and method, a recording medium, and a program, and more particularly to an imaging apparatus and method, a recording medium, and an imaging apparatus that can simultaneously acquire an image and an optical signal without degrading the resolution of the image. Regarding the program.

  Conventionally, a two-dimensional array sensor having an imaging function of an image sensor and an optical communication function by each pixel constituting the sensor has been proposed, and a new application using the two-dimensional array sensor is proposed. Has also been proposed (see Patent Document 1).

  The principle of the two-dimensional array sensor will be described with reference to FIG. In the example of FIG. 1, a camera 1 having a two-dimensional array of sensors 11 and a lens 12, and a subject 2-1 having emitting light sources 3-1 and 3-2 such as LEDs (light-emitting diodes), respectively. And 2-2 are shown.

  The sensor 11 includes a plurality of light receiving pixels (hereinafter also simply referred to as pixels) 31-n. The pixel 31-n (for example, the pixel 31-1 and the pixel 31-2) of the sensor 11 receives images of the subjects 2-1 and 2-2 existing within the field angle boundary 21 of the sensor 11 via the lens 12. While acquiring, the communication information encoded by the blink patterns 22-1 and 22-2 of the transmission light sources 3-1 and 3-2 is detected.

  That is, in the transmission light sources 3-1 and 3-2, for example, an LED light source as shown in FIG. 2 is blinked at 4 KHz, and a data code of about 8 bits is put on the blinking pattern at a certain interval. Let it emit light continuously. On the other hand, the sensor 11 performs a frame operation at about 12 KHz, which is about three times the blinking of the LED, and each light receiving pixel of the sensor 11 performs a blinking signal sampling operation, which position on the screen It is recognized whether such communication information is being emitted.

  Such a device architecture constituting the sensor 11 is also proposed in Patent Documents 2 to 4 previously filed by the present applicant. For example, Patent Documents 2 and 3 describe a sensor configured as shown in FIG.

  The sensor 11 shown in FIG. 3 includes a pixel array 31, a V (Vertical) decoder 32, an H (Horizontal) decoder 33, an analog memory array 34, a memory V decoder 35, a comparator 36, and a memory H decoder 37. .

  The pixel array 31 includes a plurality of pixels (pixels) 31-n that detect light (corresponding to the pixels 31-1 and 31-2 in FIG. 1) arranged in a two-dimensional matrix in the matrix direction. A signal transmitted from the pixel 31-n is transmitted through a signal line (vertical signal line) 38 running in the vertical direction. The V decoder 32 and the H decoder 33 control scanning (sweep) of the pixels 31-n of the pixel array 31.

  The analog memory array 34 temporarily stores the signal from the pixel array 31 in the memory cell 34-m. The memory V decoder 35 and the memory H decoder 37 control scanning (sweep) of the memory cell 34-m of the analog memory array 34. The comparator 36 compares the data extracted from the memory cell 34 -m of the analog memory array 34.

  Here, the sensor 11 has two operation modes of an image acquisition mode for acquiring an image and an optical communication mode for performing optical communication.

  In the image acquisition mode, the optical signal collected on the image array 31 is converted into an electrical signal for each pixel 31-n, and the image signal is sequentially transmitted from the H decoder 33 on the sensor 11 by the sweep operation by the V decoder 32. Is output.

  On the other hand, in the optical communication mode, an optical signal received by the same pixel array 31 is converted into an electric signal and then written into the memory cell 34-m of the lower analog memory array 34 and is already stored in the memory by the comparator 36. The magnitude of the signal at the previous frame operation stored in the cell 34-m is compared. By this comparison operation, a rising edge or a falling edge of light flashing is detected, and the detection result (also referred to as an optical signal detection result) is sequentially output from the memory H decoder 37 as binary data.

  These image acquisition mode and optical communication mode are switched in the sensor 11 at a cycle of about 15 to 60 Hz as shown in FIG. In the example of FIG. 4, t represents the passage of time.

  That is, in the image acquisition mode, the sensor 11 performs charge accumulation by light and a sweep operation of the pixel array 31 during one cycle (for example, 1/30 sec), and sequentially outputs image signals. On the other hand, in the optical communication mode, the sensor 11 performs a sweep operation of the high-speed pixel array 31 of 10,000 to several hundred thousand fps (for example, 12 kfps) during one cycle, and sequentially outputs binary data.

  Returning to FIG. 1, an example of an application using the sensor 11 configured as described above will be further described.

  For example, as information related to the subject 2-2, a transmission light source 3-2 that transmits a blinking pattern 22-2 on which data such as a website address (for example, URL (uniform resource locator)) is placed is provided. The subject 2-2 is imaged by the camera 1 provided with the two-dimensional array of sensors 11. In addition, the camera 1 can also be comprised with a portable terminal, a mobile telephone, etc.

  The camera 1 acquires an image of the subject 2-2 and detects communication information encoded by the blinking pattern 22-2 of the transmission light source 3-2. Then, the camera 1 displays the acquired image of the subject 2-2 on the monitor 42 (FIG. 5) and superimposes the address obtained from the communication information on the image 51 displayed on the monitor 42 as the icon 52. indicate.

  As a result, as shown in FIG. 5, the user can connect the camera 1 to a Web site related to the subject 2-2 by simply pointing the icon 52 displayed on the monitor 42.

  FIG. 5 is a block diagram illustrating a configuration example of the camera 1 that realizes the application.

  In the example of FIG. 5, the camera 1 includes the sensor 11 having the configuration of FIG. 3, a signal processor 41, and a monitor 42.

  The signal processor 41 captures the image signal from the sensor 11 in the image acquisition mode, performs A / D (Analog to Digital) conversion, and then performs processing for image generation. In the optical communication mode, the signal processor 41 outputs the optical signal detection result. Capture and decode this signal.

  Thereafter, the signal processor 41 combines these pieces of information, for example, is obtained from the communication information on the image 51 in which the subject 2-2 (transmitting light source 3-2) is captured, and is related to the subject 2-2. A process of superimposing icons 52 representing information on the monitor 42 is performed.

  As a result, an icon 52 representing information related to the subject 2-2 is superimposed on the image 51 of the subject 2-2 (transmitting light source 3-2) and displayed on the monitor 42.

  The camera 1 that realizes the above applications can be configured with substantially the same size and cost as a camera equipped with a normal image sensor because one sensor has both functions of normal image acquisition and optical communication. .

International Publication No. 03/036829 Pamphlet JP 2003-169251 A JP 2005-278038 A JP 2005-295181 A

  However, in this camera 1, since the image and the optical signal are received by the same pixel array 31, it is necessary to switch between the respective modes, and it is difficult to simultaneously acquire the image and the optical signal.

  On the other hand, Patent Document 4 proposes that a pixel for acquiring an image and a pixel for optical communication are separated and operated independently in the pixel array of the sensor. However, with this method, even if simultaneous acquisition is possible, image information corresponding to the pixels for optical communication is lost, resulting in degradation in resolution of the acquired image.

  For this reason, even if processing using the acquired image and communication information is performed, an image with degraded resolution must be used, which affects the accuracy of subsequent processing.

  In general, image sensors tend to try to reduce the pixel size due to the recent trend of increasing the number of pixels and reducing the size, but high-speed scanning (sweep) operation is required for small pixels. However, it is difficult to obtain sufficient detection sensitivity in an optical communication mode that requires a large number of pixels, so that it has been impossible to use an image sensor with a large number of pixels and a small size.

  The present invention has been made in view of such a situation, and makes it possible to simultaneously acquire an image and an optical signal without degrading the resolution of the image.

  An imaging apparatus according to an aspect of the present invention includes an image sensor that captures an image of a subject, an image sensor that inputs an image signal corresponding to the subject including a transmission light source that transmits an optical signal encoded with information, and the image sensor An optical communication sensor, which is arranged with a substantially angle of view at a close position and receives the optical signal transmitted from the transmission light source, associates a pixel constituting the image sensor and a pixel constituting the optical communication sensor. And a signal processing means for performing processing using information obtained by decoding the image signal input by the image sensor and the optical signal received by the optical communication sensor.

  The information may be information regarding the subject.

  The image processing device further includes display means for displaying an image corresponding to the image signal, and the signal processing means includes image generation means for generating an image corresponding to information obtained by decoding the optical signal, pixels constituting the image sensor, and Image superimposing means for performing processing for associating the pixels constituting the optical communication sensor and superimposing the image signal of the image generated by the image generating means on the image signal input by the image sensor; The display means can display an image corresponding to the image signal superimposed by the signal processing means.

  The signal processing means corresponds to an image recognition means for recognizing a subject from an image corresponding to the image signal input by the image sensor, a pixel constituting the image sensor, and a pixel constituting the optical communication sensor. In addition, there may be provided transmission source setting means for performing processing for setting a subject recognized by the image recognition means as a transmission source of information obtained by decoding the optical signal.

  Based on a first optical system driving unit that drives the optical system of the image sensor, a second optical system driving unit that drives the optical system of the optical communication sensor, and a zoom value corresponding to a user operation, An optical system of the image sensor and zoom control means for controlling the first and second optical system driving means may be further provided so that the optical system of the optical communication sensor is interlocked.

  An optical system driving unit that drives the optical system of the image sensor; and a zoom control unit that controls driving of the optical system by the optical system driving unit based on a zoom value corresponding to a user operation. The processing unit includes a corresponding pixel calculating unit that calculates a corresponding pixel of the optical communication sensor corresponding to a pixel of the image sensor based on a zoom value corresponding to a user operation, and the light calculated by the corresponding pixel calculating unit Based on the corresponding pixel of the communication sensor, the pixel constituting the image sensor and the pixel constituting the optical communication sensor are associated with each other, and the image signal input by the image sensor and received by the optical communication sensor It is possible to perform processing using information obtained by decoding the optical signal.

  The signal processing means corresponds to a pixel detection means for detecting a pixel corresponding to the transmission light source from an image corresponding to the image signal input by the image sensor, and to the transmission light source detected by the pixel detection means. And correction amount calculation means for calculating a correction amount of pixel shift based on the pixel of the optical communication sensor that has received the optical signal, and correction of the pixel shift calculated by the correction amount calculation means Based on the quantity, the image signal inputted by the image sensor and the optical signal received by the optical communication sensor are associated with the pixels constituting the image sensor and the pixels constituting the optical communication sensor. Processing using information obtained by decoding the signal can be performed.

  An image processing method according to an aspect of the present invention is an image processing method of an imaging apparatus that captures an image of an image sensor that inputs an image signal corresponding to the object including a transmission light source that transmits an optical signal encoded with information. The image is configured by associating a pixel constituting the image sensor with a pixel constituting an optical communication sensor that is disposed at a position close to the image sensor with a substantially angle of view and receives the optical signal transmitted from the transmission light source. And performing a process using information obtained by decoding the image signal input by the sensor and the optical signal received by the optical communication sensor.

  A program according to one aspect of the present invention is a program that causes an imaging device that captures an image of an object to perform image processing, and inputs an image signal corresponding to the object including a transmission light source that transmits an optical signal in which information is encoded. Associating the pixels constituting the image sensor with the pixels constituting the optical communication sensor that is arranged with a substantially angle of view at a position close to the image sensor and that receives the optical signal transmitted by the transmitting light source. Performing a process using information obtained by decoding the image signal input by the image sensor and the optical signal received by the optical communication sensor.

  A recording medium on which a program according to one aspect of the present invention is recorded is a program that causes an imaging device that captures an image of an object to perform image processing, and is provided on an object including a transmission light source that transmits an optical signal in which information is encoded. A pixel constituting an image sensor that inputs a corresponding image signal and an optical communication sensor that receives the optical signal transmitted from the transmitting light source are arranged at a position close to the image sensor with a substantially angle of view. And performing processing using information obtained by decoding the image signal input by the image sensor and the optical signal received by the optical communication sensor in association with pixels.

  In one aspect of the present invention, pixels constituting an image sensor that inputs an image signal corresponding to a subject including a light source that transmits an optical signal in which information is encoded, and a schematic image at a position close to the image sensor are provided. The image signal input by the image sensor and the optical communication sensor are associated with the pixels that constitute the optical communication sensor that is arranged at an angle and receives the optical signal transmitted from the light source. Processing using information obtained by decoding the received optical signal is performed.

  According to the present invention, an optical signal and an image of a light source that transmits the optical signal can be easily acquired simultaneously without degrading the resolution of the image.

  Embodiments of the present invention will be described below. Correspondences between constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  An imaging apparatus according to one aspect of the present invention (for example, the mobile phone 71 in FIG. 6) is a transmission light source (for example, the transmission light source 3- 1) and an image sensor (for example, the image sensor 122 in FIG. 7) for inputting an image signal corresponding to the subject (for example, the subject 2-1 in FIG. 6), and a substantially angle of view at a position close to the image sensor. The optical communication sensor (for example, the optical communication sensor 121 in FIG. 7) that receives the optical signal transmitted from the transmission light source, the pixels that constitute the image sensor, and the pixels that constitute the optical communication sensor And a signal that performs processing using information obtained by decoding the image signal input by the image sensor and the optical signal received by the optical communication sensor. Comprising processing means (e.g., an application processor 213 of FIG. 11) and.

  The image processing device further includes a display unit (for example, the monitor 82 in FIG. 6) that displays an image corresponding to the image signal, and the signal processing unit generates an image corresponding to information obtained by decoding the optical signal (for example, the monitor 82). , The image generation unit 232 in FIG. 11, the pixels constituting the image sensor and the pixels constituting the optical communication sensor are associated with each other, and the image generation means is connected to the image signal input by the image sensor. Image superimposing means (for example, the image superimposing unit 231 in FIG. 11) that performs processing for superimposing the image signal of the image generated by the image processing unit, and the display means corresponds to the image signal superimposed by the signal processing means. The image to be displayed can be displayed.

  The signal processing means includes an image recognition means (for example, an image recognition unit 341 in FIG. 22) for recognizing a subject from an image corresponding to the image signal input by the image sensor, and pixels constituting the image sensor. Source setting means (for example, FIG. 5) that performs processing for associating with the pixels constituting the optical communication sensor and setting the subject recognized by the image recognition means as the source of information obtained by decoding the optical signal. 22 information processing units 342).

  First optical system driving means (for example, the zoom driving unit 251-2 in FIG. 14) for driving the optical system of the image sensor (for example, the lens 112-2 in FIG. 14), and the optical system ( For example, the image sensor based on the second optical system driving means (for example, the zoom driving unit 251-1 in FIG. 14) for driving the lens 112-1 in FIG. 14 and the zoom value corresponding to the user's operation. Zoom control means (for example, the zoom control signal output unit 261 in FIG. 14) for controlling the first and second optical system drive means so that the optical system of the optical communication sensor is interlocked. Furthermore, it can be provided.

  Based on optical system driving means (for example, zoom driving unit 251-2 in FIG. 16) for driving the optical system of the image sensor (for example, lens 112-2 in FIG. 16) and a zoom value corresponding to a user operation. Zoom control means (for example, zoom control signal output unit 261 in FIG. 16) for controlling driving of the optical system by the optical system drive means, and the signal processing means adjusts the zoom value corresponding to the user's operation. And a corresponding pixel calculating unit (for example, corresponding pixel conversion unit 301 in FIG. 16) that calculates a corresponding pixel of the optical communication sensor corresponding to a pixel of the image sensor based on the optical communication calculated by the corresponding pixel calculating unit. Based on the corresponding pixel of the sensor, the pixel constituting the image sensor and the pixel constituting the optical communication sensor are associated with each other by the image sensor. The inputted image signal, and the information obtained by decoding the optical signal received by the optical communication sensor processing can be carried out using.

  The signal processing means includes a pixel detection means (for example, the image recognition unit 331 in FIG. 20) for detecting a pixel corresponding to the transmission light source from an image corresponding to the image signal input by the image sensor, and the pixel A correction amount calculation unit (for example, a shift in FIG. 20) that calculates a correction amount of a pixel shift based on the pixel corresponding to the transmission light source detected by the detection unit and the pixel of the optical communication sensor that has received the optical signal. A correction amount calculation unit 332), and the pixels constituting the image sensor and the pixels constituting the optical communication sensor are associated with each other based on the correction amount of the pixel shift calculated by the correction amount calculation unit. And processing using information obtained by decoding the image signal input by the image sensor and the optical signal received by the optical communication sensor. It can be.

  An image processing method or program according to one aspect of the present invention is an image processing method or program for an imaging device that captures an image of an object, and inputs an image signal corresponding to the object including a transmission light source that transmits an optical signal encoded with information. The pixels constituting the image sensor and the pixels constituting the optical communication sensor that is arranged with a substantially angle of view at a position close to the image sensor and that receives the optical signal transmitted from the light source. Then, a process using information obtained by decoding the image signal input by the image sensor and the optical signal received by the optical communication sensor is performed (for example, step S14 in FIG. 13).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 6 is a diagram showing an external configuration of a mobile phone as an imaging apparatus to which the present invention is applied. In the example of FIG. 6, the subjects 2-1 and 2-2 provided with the transmission light sources 3-1 and 3-2, respectively, are the same as those in the example of FIG. Since the detailed description will be repeated, it will be omitted as appropriate.

  A cellular phone 71 shown in FIG. 6 includes a camera unit 81, a monitor 82, and an operation input unit 83, and includes transmission light sources 3-1 and 3-2 such as LEDs (light-emitting diodes), respectively. By imaging the subject 2-1 and the subject 2-2 with the camera unit 81, the image signal obtained by the imaging and the communication information from the transmission light source 3-1 or 3-2 are acquired, and the acquired image signal is supported. The images 92-1 and 92-2 based on the communication information are superimposed on the image 91 to be displayed and displayed on the monitor 82.

  The blinking pattern 22-1 transmitted from the transmission light source 3-1 describes the website address (for example, URL (uniform resource locator)) and the subject 2-1, which are information related to the subject 2-1. The blinking pattern 22-2 transmitted by the transmission light source 3-2 is information related to the subject 2-2, for example, a URL (for example, URL) (uniform resource locator)) and a sentence describing the subject 2-2 are encoded as communication information.

  As shown in FIG. 7, the camera unit 81 includes a camera board (substrate) 111 on which the optical communication sensor 121 and the image sensor 122 are arranged, and the optical communication sensor 121 and the image sensor 122 for focusing. Each optical system (for example, lenses 112-1 and 112-2) incorporated in the front surface is configured.

  The optical communication sensor 121 is a two-dimensional array sensor composed of an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor). In other words, the CMOS image sensor generally has a higher driving speed than the CCD (Charge Coupled Device) image sensor, so even if the transmitting light source is driven at a relatively high frequency on the information transmission side, the blinking pattern can be read. Therefore, the optical communication sensor 121 is preferably composed of a CMOS image sensor.

  The image sensor 122 is, for example, a two-dimensional array sensor composed of an image sensor such as a CCD or a CMOS. That is, the optical communication sensor 121 and the image sensor 122 do not need to be configured with the same image sensor.

  The vertical and horizontal dotted lines shown in the optical communication sensor 121 and the image sensor 122 in the example of FIG. 7 indicate the vertical (V (Vertical) direction) center position and the horizontal (H (Horizontal) direction) center position of each sensor. The optical communication sensor 121 and the image sensor 122 are represented by a distance between their intersections (the center point of each sensor, that is, the point through which the optical axis C1 or C2 in FIG. It is arranged on the same camera board 111 so that d is several mm to several cm.

  The optical communication sensor 121 includes a plurality of light receiving pixels (hereinafter also simply referred to as “pixels”), as shown in the light receiving pixels 121-1 and 121-2, for example. The light receiving pixel 121-1 and the pixel 121-2 of the optical communication sensor 121 are connected to the subjects 2-1 and 2-2 existing within the field angle boundary 131 (dashed line) of the optical communication sensor 121 via the lens 112-1. The communication information encoded by the blinking patterns 22-1 and 22-2 of the transmission light sources 3-1 and 3-2 included in is detected. That is, the optical communication sensor 121 has basically the same configuration as the sensor 11 of FIG. 1 in the optical communication mode.

  Specifically, as described above with reference to FIG. 2, the information transmission side blinks the transmission light sources 3-1 and 3-2 at 4 KHz, puts a data code of about 8 bits on the blinking pattern, It emits light continuously at a certain interval. Therefore, the optical communication sensor 121 performs a frame operation at about 12 KHz, which is about three times the blinking of the transmission light sources 3-1 and 3-2, and each pixel of the optical communication sensor 121 performs a blinking signal sampling operation. It is detected what kind of communication information is emitted at which position on the screen.

  The image sensor 122 is also composed of a plurality of pixels (not shown), and the pixels of the image sensor 122 exist within the angle of view boundary 132 (solid line) of the image sensor 122 via the lens 112-2. Image signals corresponding to the images of the subjects 2-1 and 2-2 provided with 3-1 and 3-2, respectively, are acquired. That is, the image sensor 122 has basically the same configuration as the sensor 11 of FIG. 1 in the image acquisition mode.

  Returning to FIG. 6, the mobile phone 71 is configured such that the upper housing and the lower housing can be folded by a hinge (not shown).

  The monitor 82 is composed of an LCD (Liquid Crystal Display) or the like, and is provided on the side of the upper casing of the cellular phone 71 that faces the lower casing when the cellular phone 71 is folded. The monitor 82 displays a screen in which images 92-1 and 92-2 based on communication information received by the optical communication sensor 121 are superimposed on an image 91 corresponding to the image signal acquired by the image sensor 122.

  In the case of the example in FIG. 6, a balloon-type icon in which the name and description of the subject 2-1 are written on the monitor 82 as the image 92-1 is emitted from the transmission light source 3-1. The image 91 is displayed on the image 91 as if a balloon-type icon in which the name and description of the subject 2-2 are written as a balloon from the transmission light source 3-2. It is arranged and displayed above.

  That is, the user simply images the subjects 2-1 and 2-2 provided with the transmission light sources 3-1 and 3-2 using the mobile phone 71, and the subject 2-1 and 2 are displayed on the monitor 82. -2 is displayed, and the user can know the subjects 2-1 and 2-2 in detail.

  Then, in response to the user operating a button or the like constituting the operation input unit 83 and selecting the image 92-1 or 92-2 displayed on the monitor 82, the cellular phone 71 selects the selected image. An image corresponding to further detailed description information related to 92-1 (that is, the subject 2-1) is displayed on the monitor 82, or related to the selected image 92-1 (that is, the subject 2-1). You can also connect to a website that you have created.

  The operation input unit 83 includes buttons and the like. For example, the operation input unit 83 is provided on the side of the lower casing of the mobile phone 71 that faces the monitor 82 provided in the upper casing when the mobile phone 71 is folded. .

  Hereinafter, when it is not necessary to individually distinguish the subjects 2-1 and 2-2, the transmission light sources 3-1 and 3-2, and the flashing patterns 22-1 and 22-2, the subject 2, the transmission The light source 3 and the blinking pattern 22 are appropriately referred to.

  In the example of FIG. 6, the image 92-1 or 92-2 is a balloon-type icon, but is not limited to a balloon-type icon, and other icons may be used, and characters and URLs themselves may be displayed. It may be.

  FIG. 8 is a diagram illustrating the relationship between the field angle and the optical axis of the optical communication sensor 121 and the image sensor 122 in the camera unit 81.

  The optical communication sensor 121 and the image sensor 122 are arranged on the camera board 111 so that the optical axes C1 and C2 passing through the center point of each sensor are parallel to each other with a distance d between the sensors. In the camera unit 81, the field angle α1 of the optical communication sensor 121 and the field angle α2 of the image sensor 122 are the same in the vertical and horizontal directions (that is, the V direction and the H direction on the sensor surface). The distances from the camera board 111 to the lens surfaces L1 and L2 of the lenses 112-1 and 112-2 (ie, focal lengths f1 and f2) are adjusted.

  In other words, in the example of FIG. 8, the size of the projection area formed by the field angle boundary 131 of the optical communication sensor 121 and the size of the projection area formed by the field angle boundary 132 of the image sensor 122 are the same in the vertical and horizontal directions.

  In the example of FIG. 8, an example in which the optical communication sensor 121 and the image sensor 122 are arranged side by side in the H direction is shown, and the angles of view α <b> 1 and α <b> 2 are the H of the optical communication sensor 121 and the image sensor 122. It represents the angle of view of the direction.

  Further, the optical communication sensor 121 and the image sensor 122 do not need to be the same size, and the example of FIG. 8 shows a case where the size of the image sensor 122 is larger than that of the optical communication sensor 121. That is, the focal lengths f1 and f2 in the camera unit 81 in FIG. 8 satisfy f1 <f2 so that the field angle α1 of the optical communication sensor 121 and the field angle α2 of the image sensor 122 are the same in the vertical and horizontal directions. It has been adjusted.

  Pixels that receive light from the subject (transmitting light source) in the optical communication sensor 121 and the image sensor 122 arranged on the camera board 111 and adjusted in the optical system as described above will be described with reference to FIG. . Note that FIG. 9 is a conceptual diagram, and in the example of FIG. 9, for convenience of explanation, geometric relationships such as sensor size are not shown very accurately.

  In the example of FIG. 9, the lens surface L (in this case, L = L1 = L2) within the respective view angle boundaries 131 and 132 of the optical communication sensor 121 and the image sensor 122 that are arranged apart by the inter-sensor distance d. The projection area of each sensor is shown at distances D1 to D3 assuming that the distance from) is far from the vicinity.

  The dotted lines indicating the positions of the distances D1 to D3 represent the projection areas of the optical communication sensor 121 at the distances D1 to D3, and the white circled numbers shown on the dotted lines are the angle-of-view boundaries of the optical communication sensor 121. The number from the left end of the pixel in 131 is represented, and the vertical line shown above the dotted line represents the pixel projection boundary of the optical communication sensor 121. That is, between the two pixel projection boundaries on the dotted line indicating the position of each distance D1 to D3 represents the pixel projection size of the pixel of the number indicated by the white circled number at each distance D1 to D3 of the optical communication sensor 121. Yes.

  Similarly, the dotted lines indicating the positions of the distances D1 to D3 indicate the projection area of the image sensor 122 at the distances D1 to D3, and the black circled numbers below the dotted lines are the view angle boundaries 132 of the image sensor 122. The vertical line shown below the dotted line represents the pixel projection boundary of the image sensor 122. That is, between the two pixel projection boundaries below the dotted line indicating the position of each distance D1 to D3 represents the projection size of the pixel of the number indicated by the black circled number at each distance D1 to D3 of the image sensor 122.

  That is, the image of the subject existing in the projection area at each distance D1 to D3 is formed on that pixel on each sensor.

  The projection size of one pixel is determined only by the number of pixels (resolution) of the sensor, and does not depend on the size of the pixel itself on the sensor or the size of the sensor itself. In the example of FIG. 9, the optical communication sensor 121 and the image sensor 122 have the same number of pixels (resolution), and the projection size of each pixel is also the same.

  In this case, when considering the light receiving pixels of the subject (transmitting light source) on the line of sight 161 starting from the image sensor 122, the displacement of the angle of view between the optical communication sensor 121 and the image sensor 122 (that is, the distance D1 in the vicinity) , The transmission light source P1 is received by the first pixel from the left end of the view angle boundary 132 of the image sensor 122 (black circled number 1) and the left end of the view angle boundary 131 of the optical communication sensor 121. To the second pixel (white circled number 2).

  At a distance D2 farther than the distance D1, the transmission light source P2 is received by the first pixel from the left end of the field angle boundary 132 of the image sensor 122 (black circled numeral 1), and the field angle of the optical communication sensor 121. Light is received by both the first and second pixels (white circled numbers 1 and 2) from the left end of the boundary 131.

  Furthermore, at a distance D3 farther than the distance D2, light is received by the first pixel from the left end of the view angle boundary 132 of the image sensor 122 (black circled numeral 1) and from the left end of the view angle boundary 131 of the optical communication sensor 121. Light is received by the first pixel (the white circled number 1). That is, at a sufficiently long distance, both the optical communication sensor 121 and the image sensor 122 can receive the light source from the first pixel from the left from the respective field angle boundaries.

  As described above, when an attempt is made to associate the image signal acquired by the image sensor 122 with the communication information acquired by the optical communication sensor 121, the projection size of the pixel is at a distance (for example, the distance D1) that is substantially equivalent to the inter-sensor distance d. However, if the subject is located far away from the inter-sensor distance d, the pixel position will not be displaced.

  Therefore, for a subject located far enough to ignore the pixel projection size (that is, the inter-sensor distance d), it can be regarded as if image acquisition and optical communication are being performed by one sensor. .

  Furthermore, a specific description will be given with reference to FIG.

  In the example of FIG. 10, an optical communication sensor 121 and an image sensor 122 having a resolution of QVGA (Quarter Video Graphics Array) (that is, 320 × 240 pixels) are disposed on the camera board 111 with a sensor distance d = 10 mm. An example in which the optical system is adjusted with the angles of view α1 and α2 = 60 degrees of the communication sensor 121 and the image sensor 122 is shown.

  In this case, considering the light receiving pixel of the subject (transmitting light source) on the line of sight 161 starting from the image sensor 122, the transmitting light source Q1 is the image sensor 122 at a distance D = 0.35 m from the lens surface L (FIG. 9). Is received by the N = Kth pixel from the left end of the angle-of-view boundary 132, and received by the N = K + 8th pixel from the left end of the angle-of-view boundary 131 of the optical communication sensor 121. Therefore, at the distance D = 0.35 m, the light receiving pixels of the optical communication sensor 121 and the image sensor 122 are shifted by QVGA 8 pixels.

  At a distance D = 0.7 m from the lens surface L, the transmission light source Q2 is received by the N = K-th pixel from the left end of the field angle boundary 132 of the image sensor 122, and the left end of the field angle boundary 131 of the optical communication sensor 121. To N = K + 4th pixel. Therefore, at the distance D = 0.35 m, the light receiving pixels of the optical communication sensor 121 and the image sensor 122 are displaced by 4 QVGA pixels.

  At a distance D = 1.4 m from the lens surface L, the transmission light source Q3 is received by the N = K-th pixel from the left end of the view angle boundary 132 of the image sensor 122, and the left end of the view angle boundary 131 of the optical communication sensor 121. To N = K + the second pixel. Therefore, at the distance D = 1.4 m, the light receiving pixels of the optical communication sensor 121 and the image sensor 122 are displaced by QVGA 2 pixels.

  At a distance D = 2.8 m from the lens surface L, the transmission light source Q4 is received by the N = K-th pixel from the left end of the view angle boundary 132 of the image sensor 122, and the left end of the view angle boundary 131 of the optical communication sensor 121. To N = K + 1th pixel. Therefore, at the distance D = 2.8 m, the light receiving pixels of the optical communication sensor 121 and the image sensor 122 are shifted by QVGA 1 pixel.

  At an infinite distance from the distance D from the lens surface L = 2.8 m, the transmission light source Q5 is received by the N = Kth pixel from the left end of the field angle boundary 132 of the image sensor 122 and the field angle of the optical communication sensor 121. Light is received by the N = Kth pixel from the left end of the boundary 131. Therefore, at an infinite distance from the distance D = 2.8 m, the light receiving pixels of the optical communication sensor 121 and the image sensor 122 are not displaced by one QVGA pixel.

  As described above, when the distance D is 2.8 m or more, the generated pixel shift is equal to or less than one QVGA pixel, and the pixels of the optical communication sensor 121 and the image sensor 122 are considered to correspond approximately one to one. be able to.

  Thus, in the case of the cellular phone 71 in which the optical communication sensor 121 and the image sensor 122 are separately provided, if it is QVGA, as long as the distance D = 2.8 m or more, image acquisition and optical communication are executed by one sensor. Can be regarded as if

  In addition, since the image resolution does not deteriorate, for example, as described above with reference to FIG. 6, the image is acquired by the optical communication sensor 121 on the image 91 corresponding to the image signal acquired by the image sensor 122. Even if the images 92-1 and 92-2 based on the communication information are superimposed and displayed on the monitor 82, the position of the image 91 corresponding to the image signal and the positions of the images 92-1 and 92-2 based on the communication information are Depending on the distance of the subject, it can be matched almost accurately.

  As a result, even if a plurality of subjects with a transmission light source are imaged and an image of the subject with the transmission light source and an image based on communication information are superimposed and displayed on a small monitor 82 like the mobile phone 71, The position of the transmission light source in the image and, for example, the position of the image based on the communication information (the balloon-shaped icon balloon in which the name and description of the subject are described) substantially match, so the user viewed the monitor 82 This makes it possible to definitely recognize which icon belongs to which transmission light source (subject).

  FIG. 11 is a block diagram showing an example of the internal configuration of the mobile phone shown in FIG.

  In the example of FIG. 11, the mobile phone 71 includes an image signal processing unit 211, a decoding processing unit 212, and an application processor 213 in addition to the camera unit 81, the monitor 82, and the operation input unit 83 of FIG. 6. Yes.

  In the camera unit 81 of FIG. 11, an infrared cut filter 221 for maintaining image quality is disposed between the image sensor 122 and the lens 112-2.

  The image signal processing unit 211 and the decoding processing unit 212 are configured by a DSP (Digital Signal Processor) or the like. The image signal processing unit 211 performs color signal processing, gamma correction, AWB (auto white balance), and the like on the normal image sensor signal processing for the raw image signal (image RAW signal) from the image sensor 122. Perform processing such as automatic exposure compensation. The output from the image sensor 122 may be digital or analog. In the case of analog output, an A / D (Analog to Digital) conversion unit is built in the image signal processing unit 211.

  The decode processing unit 212 performs a process of decoding a binary signal (that is, a light blinking detection signal) indicating whether or not light blinking is detected from the optical communication sensor 121. A specific description will be given with reference to FIG.

  FIG. 12 schematically shows the structure of a data frame transmitted as an optical signal emitted from the transmission light source 3, that is, a blinking pattern.

  One data frame includes an 8 bit start bit, an 8 to 128 bit address, an arbitrary length of communication information (data), a 16 bit error detection bit, and an 8 bit bit. Of stop bits.

  The decode processing unit 212 detects a start bit indicating the head of the signal, and subsequently reads a signal subsequent to the pixel from which the start bit is detected, thereby decoding the binary signal.

  The application processor 213 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and responds to user operations input via the operation input unit 83. Each part of the cellular phone 71 is controlled by executing a signal to be executed and various programs stored in a built-in memory.

  That is, the application processor 213 supplies a synchronization signal (or clock signal) to the image signal processing unit 211 and the decoding processing unit 212.

  The image signal processing unit 211 generates a drive signal (timing pulse) for driving the image sensor 122 from the synchronization signal, and supplies the synchronization signal and the drive signal to the image sensor 122. The decode processing unit 212 A drive signal (timing pulse) for driving the optical communication sensor 121 is generated from the synchronization signal, and the synchronization signal and the drive signal are supplied to the optical communication sensor 121.

  In addition, the application processor 213 includes a real-time image signal from the image signal processing unit 211, information on a pixel that has received the optical signal from the decoding processing unit 212 (hereinafter referred to as reception pixel information), and decoding. The communication information obtained as a result is input. The application processor 213 realizes various functions by associating and merging the image signal from the image sensor 122 and the communication information from the optical communication sensor 121.

  Specifically, in the case of the example in FIG. 11, the application processor 213 executes a predetermined program to internally configure the image superimposing unit 231 and the image generating unit 232, and the pixels configuring the image sensor 122. The pixels constituting the optical communication sensor 121 are associated one-to-one, and an image corresponding to the image signal from the image sensor 122 is obtained using the image signal from the image sensor 122 and the communication information from the optical communication sensor 121. Then, the image based on the communication information from the optical communication sensor 121 is superposed, and the superposed image is displayed on the monitor 82.

  Based on the communication information obtained as a result of the decoding, the image generation unit 232, for example, the image 92-1 which is a balloon-type icon in which the name and description of the subject 2-1 in FIG. -2 is generated as an image signal, and is output to the image superimposing unit 231 as an image signal. Information on received pixels from the decoding processing unit 212 is also supplied to the image superimposing unit 231 via the image generating unit 232.

  The image superimposing unit 231 superimposes the real-time image signal from the image signal processing unit 211 and the image signal generated by the image generation unit 232 on the basis of information on the received pixels from the decoding processing unit 212, and superimposes the image. The signal is output to the monitor 82, and an image corresponding to the superimposed image signal is displayed on the monitor 82.

  That is, the image superimposing unit 231 associates the pixels constituting the image sensor 122 with the pixels constituting the optical communication sensor 121 on a one-to-one basis, and performs optical communication on the image corresponding to the image signal from the image sensor 122. The image generated by the image generation unit 232 is superimposed on the position indicated by the information of the received pixels from the sensor 121 and displayed on the monitor 82.

  As a result, as described above with reference to FIG. 6, a balloon-type icon on which the name and description of the subject 2-1 are written as the image 92-1 is displayed on the monitor 82 from the transmission light source 3-1. A balloon-type icon that is arranged and displayed on the image 91 as if it is ballooned and in which the name and description of the subject 2-2 are written as the image 92-2 appears to be ballooned from the transmission light source 3-2. Are arranged and displayed on the image 91.

  In the above description, the example in which the optical communication sensor 121 and the image sensor 122 are arranged on the same substrate (camera mode 111) has been described. However, the optical communication sensor 121 and the image sensor 122 do not have to be on the same substrate, and two independent camera modules are adjacent to each other. It is also possible to arrange them.

  Moreover, although the case where the number of pixels (resolution) of both sensors (the optical communication sensor 121 and the image sensor 122) is the same was described, it does not necessarily need to be the same. In other words, the image sensor 122 may be configured with a sensor having 1 million pixels or more, and the optical communication sensor 121 may be configured with a low resolution sensor such as QVGA (77,000 pixels).

  However, in this case, the association between the pixels constituting the image sensor 122 and the pixels constituting the optical communication sensor 121 in the application processor 213 is performed at a ratio according to the number of pixels such as 2: 1 or 3: 1. . That is, in the application processor 213, the correspondence between the pixels constituting the image sensor 122 and the pixels constituting the optical communication sensor 121 is not limited to one-to-one, and the pixel group constituting the image sensor 122 and the optical communication sensor 121 are constituted. The corresponding pixel group is associated one-to-one.

  Next, image signal processing by the application processor 213 in FIG. 11 will be described with reference to the flowchart in FIG. This image signal process is a process that is started in response to a user operating a camera activation button or the like constituting the operation input unit 83.

  When the application processor 213 starts supplying the synchronization signal (or clock signal) to the image signal processing unit 211 and the decoding processing unit 212, the image signal processing unit 211 drives the image sensor 122 from the synchronization signal. A drive signal (timing pulse) is generated, and supply of the synchronization signal and the drive signal to the image sensor 122 is started.

  Based on the synchronization signal and the drive signal from the image signal processing unit 211, the image sensor 122 passes through the lens 112-2 to detect the subjects 2-1 and 2-2 existing in the field angle boundary 132 of the image sensor 122. An image signal corresponding to the image is acquired, and output of the acquired image signal to the image signal processing unit 211 is started.

  Therefore, the image signal processing unit 211 performs color signal processing, gamma correction, AWB (auto white balance) on the raw image signal (image RAW signal) from the image sensor 122 in the same manner as signal processing of a normal image sensor. ), Automatic exposure compensation, etc.

  Correspondingly, the image superimposing unit 231 of the application processor 213 takes in the image signal subjected to signal processing from the image signal processing unit 211 in step S11.

  On the other hand, the decode processing unit 212 generates a drive signal (timing pulse) for driving the optical communication sensor 121 from the synchronization signal from the application processor 213 and starts supplying the synchronization signal and the drive signal to the optical communication sensor 121.

  Based on the synchronization signal and the drive signal from the decoding processing unit 212, the optical communication sensor 121 passes through the lens 112-1 and the subjects 2-1 and 2-2 existing within the field angle boundary 131 of the optical communication sensor 121. Communication information encoded by the blinking patterns 22-1 and 22-2 of the transmission light sources 3-1 and 3-2 included in the information is detected, and the detected information (that is, a binary signal indicating the presence / absence of light blinking detection) is detected. A certain flashing detection signal) starts to be output to the decoding processing unit 212.

  The decode processing unit 212 performs a process of decoding the light blinking detection signal from the optical communication sensor 121. That is, the decoding processing unit 212 detects a start bit indicating the head of the light blinking detection signal, and subsequently decodes the binary signal by reading the signal after the pixel where the start bit is detected.

  In response to this, the image generation unit 232 of the application processor 213 acquires communication information obtained as a result of decoding from the decoding processing unit 212 in step S12. At this time, the image generation unit 232 also acquires information on the received pixel that has received the optical signal.

  In step S13, the image generation unit 232, for example, based on the communication information obtained as a result of decoding from the decoding processing unit 212, for example, a balloon type in which the name and description of the subject 2-1 in FIG. An image 92-1 that is an icon and an image 92-2 that is a balloon-type icon in which the name and description of the subject 2-2 are written are generated and output to the image superimposing unit 231 as an image signal. Note that information on the pixel that has received the signal from the decoding processing unit 212 is also output to the image superimposing unit 231 via the image generating unit 232.

  In step S <b> 14, the image superimposing unit 231 receives the real-time image signal from the image signal processing unit 211 and the icon image signal generated by the image generation unit 232 based on the received pixel information from the decoding processing unit 212. The superimposed image signal is output to the monitor 82.

  In step S15, the monitor 82 displays an image corresponding to the image signal from the image superimposing unit 231.

  That is, the image superimposing unit 231 corresponds to the image signal from the image sensor 122 by associating the pixels constituting the image sensor 122 and the pixels constituting the optical communication sensor 121 in one-to-one correspondence in steps S14 and S15. The image generated by the image generation unit 232 is superimposed on the position indicated by the information of the received pixel from the optical communication sensor 121 on the image and displayed on the monitor 82.

  As a result, as described above with reference to FIG. 6, a balloon-type icon on which the name and description of the subject 2-1 are written as the image 92-1 is displayed on the monitor 82 from the transmission light source 3-1. A balloon-type icon that is arranged and displayed on the image 91 as if it is ballooned and in which the name and description of the subject 2-2 are written as the image 92-2 appears to be ballooned from the transmission light source 3-2. Are arranged and displayed on the image 91.

  As described above, the sensor for optical communication and the sensor for image acquisition are configured separately, the angle of view of each sensor is matched, and they are arranged side by side by a predetermined distance, and the image sensor is an image acquisition sensor. The pixels constituting 122 and the pixels constituting the optical communication sensor 121, which is a sensor for optical communication, are associated with each other (one-to-one in the case of FIG. 13) to perform signal processing (for example, from the image signal processing unit 211). The real-time image signal and the image signal of the icon generated by the image generation unit 232 are superimposed).

  Thus, image acquisition and optical communication can be performed simultaneously without degrading the image quality, and the position of the subject (transmitting light source) on the image corresponding to the image signal from the image sensor 122 and the optical communication sensor 121 are reduced. The positions indicated by the information of the received pixels from can be substantially matched.

  Therefore, for example, when image acquisition and optical communication are performed on a plurality of subjects having a transmission light source, the correspondence between the information received by optical communication and the transmission light source on the image is substantially accurate. Therefore, the accuracy of subsequent processing (for example, the above-described superimposed display processing) can be improved.

  In general, image sensors tend to reduce the pixel size due to the recent trend of increasing the number of pixels and the size (trend). In the case of a camera having a communication mode, it has been difficult to obtain sufficient detection sensitivity in an optical communication mode that requires a high-speed scanning (sweep) operation. Further, in an ordinary image sensor, an infrared cut filter is inserted into the optical system to maintain image quality. However, when it is applied to a camera having a conventional image acquisition mode and an optical communication mode, for example, Infrared rays could not be used as a medium.

  In addition, as described above with reference to FIG. 5, when considering application development by the configuration of a camera having a conventional image acquisition mode and an optical communication mode, selection of an image acquisition format (number of pixels, sensor size), etc. The room is narrow, it is difficult to use a CCD image sensor, and image processing software used in a normal image sensor cannot be used as it is.

  However, by configuring the mobile phone 71 as described above, the image acquisition sensor does not depend on an optical communication sensor such as a CCD or a CMOS, but is used in a general image sensor or a small size. The optical communication sensor is not restricted in terms of design and function.

  Furthermore, as described above with reference to FIG. 11, since the infrared cut filter 221 for maintaining the image quality has only to be inserted into the optical system of the image sensor 122, the optical communication sensor 121 can receive infrared rays. Become. That is, it is possible to use not only visible light but also infrared rays as a communication medium for optical communication together with image maintenance.

  In addition, the image sensor 122 and the image signal processing unit 211 in the subsequent stage can use a general-purpose image sensor chip set. A general-purpose image sensor or the like can be used.

  As described above, it is possible to improve the efficiency of application development, and it is possible to reduce development costs.

  FIG. 14 is a block diagram showing another example of the internal configuration of the mobile phone shown in FIG. 14 includes the camera unit 81, the monitor 82, the operation input unit 83, the image signal processing unit 211, the decoding processing unit 212, and the application processor 213 in common with the mobile phone 71 of FIG. However, the difference is that zoom drive units 251-1 and 251-2 are added.

  The application processor 213 in FIG. 14 is common to the application processor 213 in FIG. 11 in that the application processor 213 includes an image superimposing unit 231 and an image generation unit 232, but is different in that a zoom control signal output unit 261 is added. .

  That is, the operation input unit 83 in the example of FIG. 14 inputs, for example, the zoom value of the optical zoom of the camera unit 81 to the zoom control signal output unit 261 in response to a user operation.

  The zoom control signal output unit 261 sets the zoom value input via the operation input unit 83 as the optical zoom value of the image sensor 122, and sets the set optical zoom value (control signal) to the zoom drive unit 251- 2 is supplied.

  At this time, the zoom control signal output unit 261 is input via the operation input unit 83 so as to drive the lens 112-1 of the optical communication sensor 121 in conjunction with the driving of the lens 112-2 of the image sensor 122. Based on the zoom value to be set, the optical zoom value of the optical communication sensor 121 is set, and the set optical zoom value is supplied to the zoom drive unit 251-1.

  The zoom drive unit 251-1 drives the position of the lens 112-1 that is the optical system of the optical communication sensor 121 based on the optical zoom value of the optical communication sensor 121 supplied from the zoom control signal output unit 261. The zoom drive unit 251-2 drives the position of the lens 112-2 that is the optical system of the image sensor 122 based on the optical zoom value of the image sensor 122 supplied from the zoom control signal output unit 261.

  As a result, an image signal enlarged based on the optical zoom value is input from the image sensor 122 and received from the optical communication sensor 121 by a pixel whose light receiving area is reduced based on the optical zoom value. Communication information is input.

  The image processing by the application processor 213 in FIG. 14 is basically the same processing as the image processing by the application processor 213 in FIG. 11 described above with reference to FIG.

  In the example of FIG. 14, the zoom drive unit that drives the optical system of the optical communication sensor 121 and the optical system of the image sensor 122 is configured separately as zoom drive units 251-1 and 251-2. One optical system can be configured to drive two optical systems.

  Next, with reference to FIG. 15, the pixel shift when the double optical zoom is set will be described.

  In the example of FIG. 15, as described above with reference to FIG. 10, the optical communication sensor 121 and the image sensor 122 having the QVGA resolution are disposed on the camera board 111 with the inter-sensor distance d = 10 mm. In addition, an example is shown in which the optical zoom is further set in a state where the optical system is adjusted with the field angles α1 and α2 = 60 degrees of the image sensor 122, respectively.

  That is, by setting the optical zoom of 2 times, the angles of view α1 and α2 of the optical communication sensor 121 and the image sensor 122 become 30 degrees, respectively, and the angle of view boundary 131 (one-dot chain line) of the optical communication sensor 121 is the optical communication sensor. An angle of view boundary 281 (thick alternate long and short dash line) in which the distance a to the optical axis C1 of 121 becomes a distance a / 2 that is 1/2 of the distance a, and the angle of view boundary 132 (solid line) of the image sensor 122 is A field angle boundary 282 (thick line) in which the distance b to the optical axis C2 is a distance b / 2 that is ½ of the distance b.

  In this case, at the distance D = 0.35 m from the lens surface L, the light receiving pixels of the optical communication sensor 121 and the image sensor 122 are shifted by QVGA 16 pixels (that is, twice as much as in FIG. 10). At a distance D = 0.7 m from the lens surface L, the light receiving pixels of the optical communication sensor 121 and the image sensor 122 are shifted by QVGA 8 pixels (that is, twice as much as FIG. 10). In the example of FIG. 15, for the convenience of explanation, detailed illustration in the case of the distance D = 0.35 m and the distance D = 0.7 m is omitted.

  At the distance D = 1.4 m from the lens surface L, the first pixel from the left end of the view angle boundary 282 of the image sensor 122 corresponds to the fifth pixel from the left end of the view angle boundary 281 of the optical communication sensor 121. Yes. In other words, the light receiving pixels of the optical communication sensor 121 and the image sensor 122 are shifted by QVGA 4 pixels (that is, twice as large as that in FIG. 10).

  At the distance D = 2.8 m from the lens surface L, the first pixel from the left end of the view angle boundary 282 of the image sensor 122 corresponds to the third pixel from the left end of the view angle boundary 281 of the optical communication sensor 121. Yes. That is, the light receiving pixels of the optical communication sensor 121 and the image sensor 122 are shifted by two QVGA pixels (that is, twice as many as those in FIG. 10).

  At an infinite distance from the distance D from the lens surface L = 2.8 m, the first pixel from the left end of the view angle boundary 282 of the image sensor 122 is the first pixel from the left end of the view angle boundary 281 of the optical communication sensor 121. It corresponds to. Therefore, at an infinite distance from the distance D = 2.8 m, the light receiving pixels of the optical communication sensor 121 and the image sensor 122 are not displaced by one QVGA pixel.

  As described above, with respect to the pixels that receive the same light source in the optical communication sensor 121 and the image sensor 122, when the double optical zoom is used, the pixel shift is twice that when the optical zoom is not used.

  The allowable amount of such deviation varies depending on the application for signal processing using the communication information received by the optical communication sensor 121 and the image signal captured by the image sensor 122. For example, in the application for the mobile phone 71, Since the pixel resolution is not so large and the screen of the monitor 82 is small, a deviation of several pixels is allowed.

  As described above, even when the camera unit 81 has an optical zoom function, the angle of view relationship can be maintained by linking the optical zoom with the image sensor 122 and the optical communication sensor 121. As in the case of the cellular phone 71 of FIG. 11, image acquisition and optical communication can be performed simultaneously without degrading the image quality, and the transmission of the subject on the image corresponding to the image signal from the image sensor 122 is transmitted. The position indicated by the information of the received pixel from the optical communication sensor 121 can be substantially matched.

  Thus, even when the camera unit 81 has an optical zoom function, for example, when image acquisition and optical communication are performed on a plurality of subjects having a transmission light source, information received by optical communication Since the correspondence with the transmission light source on the image can be made substantially accurate, the accuracy of the subsequent processing (for example, the above-described superimposed display processing) can be improved.

  FIG. 16 is a block diagram showing another example of the internal configuration of the mobile phone shown in FIG. 16 includes the camera unit 81, the monitor 82, the operation input unit 83, the image signal processing unit 211, the decoding processing unit 212, the application processor 213, and the zoom control unit 251-2. Although it is the same as that of the cellular phone 71, it is different in that the zoom control unit 251-1 is removed.

  In addition, the application processor 213 in FIG. 16 is common to the application processor 213 in FIG. 14 in that it includes an image superimposing unit 231, an image generating unit 232, and a zoom control signal output unit 261. The point that is added is different.

  That is, the operation input unit 83 in the example of FIG. 16 inputs, for example, the zoom value of the optical zoom of the camera unit 81 to the zoom control signal output unit 261 and the corresponding pixel conversion unit 301 in response to a user operation.

  The zoom control signal output unit 261 sets the zoom value input via the operation input unit 83 as the optical zoom value of the image sensor 122, and supplies the set optical zoom value to the zoom drive unit 251-2.

  The corresponding pixel conversion unit 301 calculates the pixel (number) of the optical communication sensor 121 corresponding to the pixel (number) of the image sensor 122 based on the zoom value input via the operation input unit 83, and the calculated correspondence. Pixel information is supplied to the image superimposing unit 231.

  The image superimposing unit 231, based on the corresponding pixel information from the corresponding pixel conversion unit 301 and the received pixel information from the decoding processing unit 212, the real-time image signal from the image signal processing unit 211, and the image generation unit 232. The superimposed image signal is output to the monitor 82, and an image corresponding to the superimposed image signal is displayed on the monitor 82.

  FIG. 17 shows an example where only the image sensor 122 in FIG. 8 described above is set to double optical zoom.

  As in the case of FIG. 8, the optical communication sensor 121 and the image sensor 122 are arranged on the camera board 111 so that the optical axes C1 and C2 passing through the center points of the sensors are parallel to each other with a distance d between the sensors. Has been. Further, in the camera unit 81, since the 2 × optical zoom is set in the image sensor 122, the camera angle 81 of the image sensor 122 is set to ½ of the field angle α 1 of the optical communication sensor 121. The distances from the board 111 to the lens surfaces L1 and L2 of the lenses 112-1 and 112-2 (ie, focal lengths f1 and f2) are adjusted.

  That is, in the example of FIG. 17, the projection area by the field angle boundary 132 of the image sensor 122 is ½ of the projection area by the field angle boundary 131 of the optical communication sensor 121.

  Accordingly, the corresponding pixel conversion unit 301 sets an area half the center of the light receiving area of the optical communication sensor 121 as an information extraction area E, and the pixels in the information extraction area E correspond to the pixels of the light receiving area of the image sensor 122. The pixel number of the optical communication sensor 121 corresponding to the pixel of the image sensor 122 is calculated.

  That is, the corresponding pixel conversion unit 301 simulates the angle of view boundary 131 of the optical communication sensor 121 in an image in which the distance a to the optical axis C1 of the optical communication sensor 121 is ½ of the distance a / 2. As a corner boundary (hereinafter also referred to as a pseudo view angle boundary) 311 (thick one-dot chain line), the pixels in the projection area formed by the view angle boundary 132 correspond to the pixels in the projection area formed by the pseudo view angle boundary 311.

  This will be specifically described with reference to FIG.

  In the example of FIG. 18, as described above with reference to FIG. 10, the optical communication sensor 121 and the image sensor 122 having the QVGA resolution are arranged on the camera board 111 with the inter-sensor distance d = 10 mm. In the state where the optical system is adjusted with the angle of view α1 and α2 = 60 degrees of the image sensor 122 and the image sensor 122, an example in which a double optical zoom is set only for the image sensor 122 is shown. .

  That is, by setting the optical zoom twice as large as that of the image sensor 122, the angle of view α2 of the image sensor 122 is 30 degrees, and the angle of view boundary 132 of the image sensor 122 has a distance of α2 In other words, the pixel projection size of the image sensor 122 is ½ of the pixel projection size of the optical communication sensor 121.

  Therefore, the corresponding pixel conversion unit 301 simulates the field angle boundary 131 of the optical communication sensor 121 in a pseudo manner so that the distance a to the optical axis C1 of the optical communication sensor 121 is a distance a / 2 that is ½ of the distance a. As the angle-of-view boundary 311, two pixels in the projection area by the angle-of-view boundary 132 of the image sensor 122 (two pixels from 1 to 320 from the left end of the angle-of-view boundary 132) are used as one pixel of the projection area by the pseudo-angle boundary 311 Corresponding to each of the 81st to 240th pixels from the left end of the view angle boundary 131.

  In this case, at the distance D = 1.4 m from the lens surface L, the first and second pixels (black circled numbers 1 and 2) from the left end of the view angle boundary 132 of the image sensor 122 are the image of the optical communication sensor 121. This corresponds to the 83rd pixel from the left end of the corner boundary 131 (white circled numeral 83). This pixel is the third pixel from the left end of the pseudo view angle boundary 311 of the optical communication sensor 121. Therefore, as in the case of the example of FIG. 15, the light receiving pixels of the optical communication sensor 121 and the image sensor 122 are shifted by 4 pixels of QVGA when the image sensor 122 is double zoomed.

  At the distance D from the lens surface L = 2.8 m, the first to fourth pixels (black circled numbers 1 to 4) from the left end of the field angle boundary 132 of the image sensor 122 are the field angle boundary 131 of the optical communication sensor 121. Corresponds to the 82nd and 83rd pixels (white circled numbers 82 and 83) from the left end. These pixels are the second and third pixels from the left end of the pseudo view angle boundary 311 of the optical communication sensor 121. Therefore, as in the case of the example of FIG. 15, the light receiving pixels of the optical communication sensor 121 and the image sensor 122 are displaced by two QVGA pixels when the image sensor 122 is double zoomed.

  When the distance D from the lens surface L is infinitely farther than 2.8 m, the first to fourth pixels (black circled numbers 1 to 4) from the left end of the field angle boundary 132 of the image sensor 122 are the optical communication sensor 121. This corresponds to the 81st to 82nd pixels (white circled numbers 81 and 82) from the left end of the view angle boundary 131. These pixels are the first and second pixels from the left end of the pseudo view angle boundary 311 of the optical communication sensor 121. Accordingly, at an infinite distance from the distance D = 2.8 m, the light receiving pixels of the optical communication sensor 121 and the image sensor 122 are not displaced by one QVGA pixel when the image sensor 122 is double zoomed.

  In the application processor 213 of FIG. 16, the pixels of the image sensor 122 and the pixels of the optical communication sensor 121 are associated with each other on a two-to-one basis. That is, when zooming is performed, the pixels constituting the image sensor 122 and the pixels constituting the optical communication sensor 121 are associated with each other at a ratio corresponding to the zoom magnification.

  As described above, the corresponding pixel conversion unit 301 uses the zoom value (in the example of FIG. 18, doubled) to project the pixels in the projection area by the pseudo view angle boundary 311 (81 to 240 from the left end of the view angle boundary 131). The second pixel) is made to correspond to the pixel in the projection area by the angle of view boundary 282 (the first to 320th pixels from the left end of the angle of view boundary 282), so when using the optical zoom function of the camera unit 81, Even if the optical zoom of the optical communication sensor 121 and the image sensor 122 is not linked, the same effect as when the optical zoom of the optical communication sensor 121 and the image sensor 122 is linked can be obtained.

  That is, even if the camera unit 81 has an optical zoom function, the optical zoom is not necessary if the optical zoom value of the image sensor 122 is known without interlocking the optical zoom between the image sensor 122 and the optical communication sensor 121. By calculating the pixel shift based on the zoom value, it is possible to maintain the field angle relationship. Therefore, as in the case of the mobile phone 71 of FIG. Optical communication can be performed at the same time, and the position of the subject (transmitting light source) on the image corresponding to the image signal from the image sensor 122 and the position indicated by the information of the received pixel from the optical communication sensor 121 are substantially matched. You can also

  Next, image signal processing by the application processor 213 in FIG. 16 will be described with reference to the flowchart in FIG. Note that steps S33 to S35 and step S37 in FIG. 19 are basically the same processes as steps S11 to S13 and step S15 in FIG. 13 and are repeated, and thus detailed description thereof will be omitted as appropriate.

  For example, the user operates a zoom button or the like constituting the operation input unit 83 in order to enlarge and display the image displayed on the monitor 82 in step S15 of FIG. The operation input unit 83 inputs a zoom value desired by the user to the zoom control signal output unit 261 and the corresponding pixel conversion unit 301 in response to a user operation.

  In step S31, the zoom control signal output unit 261 sets the zoom value input from the operation input unit 83 as the optical zoom value of the image sensor 122, and supplies the set optical zoom value to the zoom drive unit 251-2. To do.

  In response to this, the zoom drive unit 251-2 determines the position of the lens 112-2 that is the optical system of the image sensor 122 based on the optical zoom value of the image sensor 122 supplied from the zoom control signal output unit 261. To drive.

  In step S <b> 32, the corresponding pixel conversion unit 301 calculates the pixel of the optical communication sensor 121 corresponding to the pixel of the image sensor 122 based on the zoom value input from the operation input unit 83, and uses the calculated corresponding pixel information. And supplied to the image superimposing unit 231.

  On the other hand, the image sensor 122 is connected to the image sensor 122 via the lens 112-2 whose position is driven based on the optical zoom value by the zoom driving unit 251 based on the synchronization signal and the driving signal from the image signal processing unit 211. Image signals corresponding to the images of the subjects 2-1 and 2-2 existing within the view angle boundary 132 are acquired, and the acquired image signals are output to the image signal processing unit 211.

  The image signal processing unit 211 performs processing such as color signal processing, gamma correction, AWB, and automatic exposure correction on the raw image signal from the image sensor 122 in the same manner as normal image sensor signal processing. .

  In response to this, the image superimposing unit 231 of the application processor 213 takes in the image signal subjected to signal processing from the image signal processing unit 211 in step S33.

  Further, the optical communication sensor 121 receives the subjects 2-1 and 2 existing in the field angle boundary 131 of the optical communication sensor 121 via the lens 112-1, based on the synchronization signal and the drive signal from the decoding processing unit 212. -2 detects the communication information encoded by the blinking patterns 22-1 and 22-2 of the transmission light sources 3-1 and 3-2 included in -2, and outputs the light blinking detection signal to the decoding processing unit 212. .

  The decode processing unit 212 decodes a binary signal (that is, a light blinking detection signal) indicating whether or not light blinking is detected from the optical communication sensor 121.

  In response to this, the image generation unit 232 of the application processor 213 acquires communication information obtained as a result of decoding from the decoding processing unit 212 in step S34. At this time, the image generation unit 232 also acquires information on the received pixel that has received the optical signal.

  In step S35, the image generation unit 232 generates an image such as an icon based on the communication information obtained as a result of decoding from the decoding processing unit 212, and outputs the image as an image signal to the image superimposing unit 231. Note that received pixel information from the decoding processing unit 212 is also output to the image superimposing unit 231 via the image generating unit 232.

  In step S <b> 36, the image superimposing unit 231 associates the pixels constituting the image sensor 122 and the pixels constituting the optical communication sensor 121 in a two-to-one relationship with the corresponding pixel information calculated by the corresponding pixel conversion unit 301. Based on the pixel information from the decode processing unit 212, the real-time image signal from the image signal processing unit 211 and the image signal of the icon generated by the image generation unit 232 are superimposed, and the superimposed image signal is monitored 82. Output to.

  In step S <b> 37, the monitor 82 displays an image corresponding to the image signal from the image superimposing unit 231.

  As described above, even when the camera unit 81 has an optical zoom function, the optical zoom is linked by the image sensor 122 and the optical communication sensor 121, or the light based on the optical zoom of the image sensor 122 is used. Since the field angle relationship is maintained by making the pixels of the communication sensor 121 correspond to the pixels of the image sensor 122, as in the case of the mobile phone 71 of FIG. Optical communication can be performed simultaneously.

  In the above description, an effective example has been described in which the transmission light source is disposed at some distance from the two sensors (the image sensor 122 and the optical communication sensor 121) so that the pixel shift between the two sensors can be ignored. However, this is because it is assumed that the resolution of the image sensor 122 and the monitor 82 of the mobile phone 71 is not so high in many cases, and that the transmitted light source at a position away from the user is often imaged.

  Therefore, when the transmission light source is arranged in the vicinity of the two sensors, or when the resolution of the image sensor 122 and the monitor 82 is high and the pixel shift between the two sensors is conspicuous, the pixel shift between the two sensors is reduced. It is necessary to correct.

  FIG. 20 shows an example of the internal configuration of the mobile phone of FIG. 11 when correcting the pixel shift between the two sensors. In the example of FIG. 20, the camera unit 81, the monitor 82, the operation input unit 83, the image signal processing unit 211, the decoding processing unit 212, and the application processor 213 are common to the mobile phone 71 of FIG. Yes.

  The application processor 213 in FIG. 20 includes the image superimposing unit 231 and the image generation unit 232 in common with the application processor 213 in FIG. 11, but the image recognition unit 331 and the deviation correction amount calculation unit 332 are added. The difference was made.

  The image recognizing unit 331 takes in the image signal after the signal processing from the image signal processing unit 211 and performs image recognition, thereby calculating the position of the transmission light source (pixel to be imaged) and shifting the calculated position of the transmission light source. This is supplied to the correction calculation unit 332. Specifically, since the image of the transmission light source has a light luminance value higher on the peak than the surrounding image, the image recognizing unit 331 has a signal intensity specific to the luminance signal of the adjacent pixel. The pixel having a higher value is extracted by the difference calculation process, and the position of the transmission light source is calculated.

  The deviation correction calculation unit 332 acquires information on the received pixel that has received the optical signal (that is, the pixel on which the light source is imaged) from the decode processing unit 212 and compares it with the position of the light source from the image recognition unit 331. Then, the correction amount of the pixel shift obtained by the comparison is calculated, and the calculated correction amount of the pixel shift is supplied to the image superimposing unit 231.

  When there are a plurality of transmission light sources in the angle of view, the deviation correction calculation unit 332 has the closest of the positions of the plurality of transmission light sources by the optical communication sensor 121 and the positions of the plurality of transmission light sources by the image sensor 122. As a corresponding transmission light source, a pixel shift correction amount is calculated.

  The image superimposing unit 231 receives the real-time image signal from the image signal processing unit 211 and the image generation unit 232 based on the received pixel information from the decoding processing unit 212 and the pixel shift correction amount from the shift correction calculation unit 332. The superimposed image signal is output to the monitor 82, and an image corresponding to the superimposed image signal is displayed on the monitor 82.

  Next, image signal processing by the application processor 213 in FIG. 20 will be described with reference to the flowchart in FIG. Note that steps S51, S52, S55, and S57 in FIG. 21 are basically the same processes as steps S11 to S13, and S15 in FIG. 13 and are repeated, and thus detailed description thereof will be omitted as appropriate.

  Based on the synchronization signal and the drive signal from the image signal processing unit 211, the image sensor 122 passes through the lens 112-2 to detect the subjects 2-1 and 2-2 existing in the field angle boundary 132 of the image sensor 122. The image signal corresponding to the image is acquired, and the acquired image signal is output to the image signal processing unit 211. The image signal processing unit 211 is a normal image sensor for the raw image signal from the image sensor 122. Similar to the signal processing, color signal processing, gamma correction, AWB, automatic exposure correction, and other processing are performed.

  In response to this, the image superimposing unit 231 and the image recognizing unit 331 of the application processor 213 fetch the image signal subjected to signal processing from the image signal processing unit 211 in step S51.

  On the other hand, the optical communication sensor 121 receives the subjects 2-1 and 2 existing in the field angle boundary 131 of the optical communication sensor 121 via the lens 112-1, based on the synchronization signal and the drive signal from the decoding processing unit 212. -2 detects the communication information encoded by the blinking patterns 22-1 and 22-2 of the transmission light sources 3-1 and 3-2 included in -2, and outputs the light blinking detection signal to the decode processing unit 212. The decoding processing unit 212 decodes a light blinking detection signal indicating whether or not light blinking is detected from the optical communication sensor 121.

  In response to this, the image generation unit 232 of the application processor 213 acquires communication information obtained as a result of decoding from the decoding processing unit 212 in step S52. At this time, the image generation unit 232 and the deviation correction calculation unit 332 acquire information on the reception pixel that has received the optical signal.

  In step S53, the image recognition unit 331 performs image recognition using the image signal from the image signal processing unit 211, thereby calculating the position of the transmission light source (pixel to be imaged), and the calculated position of the transmission light source. Is supplied to the deviation correction calculation unit 332.

  In step S54, the deviation correction calculation unit 332 compares the received pixel (that is, the pixel on which the transmission light source is imaged) that received the optical signal from the decoding processing unit 212 with the position of the transmission light source from the image recognition unit 331. Then, the correction amount of the pixel shift obtained by the comparison is calculated, and the calculated correction amount of the pixel shift is supplied to the image superimposing unit 231.

  In step S55, the image generation unit 232 generates, for example, an icon image based on the communication information obtained as a result of decoding from the decoding processing unit 212, and outputs the image as an image signal to the image superimposing unit 231. Note that information on the received pixel that has received the signal from the decoding processing unit 212 is also supplied to the image superimposing unit 231 via the image generating unit 232.

  In step S56, the image superimposing unit 231 associates the pixels constituting the image sensor 122 and the pixels constituting the optical communication sensor 121 on a one-to-one basis, and information on the pixels from the decoding processing unit 212 and shift correction calculation. Based on the pixel shift correction amount from the unit 332, the real-time image signal from the image signal processing unit 211 and the image signal of the icon generated by the image generation unit 232 are superimposed, and the superimposed image signal is output to the monitor 82. To do.

  In step S57, the monitor 82 displays an image corresponding to the image signal from the image superimposing unit 231.

  As described above, the transmission light source (subject) is disposed in the vicinity of the two sensors, or the resolution of the image sensor 122 and the monitor 82 is high and the pixel shift between the two sensors is conspicuous. In addition, since the pixel shift between the two sensors is corrected, the correspondence between the image signal acquired by the image sensor 122 and the communication information acquired by the optical communication sensor 121 becomes more accurate.

  Accordingly, image acquisition and optical communication can be performed simultaneously without degrading the image quality, and the position of the subject (transmitting light source) on the image corresponding to the image signal from the image sensor 122 and the optical communication sensor 121 The positions indicated by the information of the received pixels can be substantially matched regardless of the distance of the subject.

  Thus, for example, when image acquisition and optical communication are performed on a plurality of subjects having a transmission light source, the correspondence between the information received by the optical communication and the transmission light source on the image is determined as the subject distance. Therefore, the accuracy of the subsequent processing (for example, the above-described superimposition display processing) can be improved.

  In the description of FIG. 20 and FIG. 21, the case where the pixels configuring the image sensor 122 and the pixels configuring the optical communication sensor 121 are associated with each other one to one has been described. Can also be applied in the case of matching at a ratio according to the number of pixels and the magnification.

  Here, in the above description, the pixels constituting the image sensor 122 and the pixels constituting the optical communication sensor 121 are associated with each other, and processing using information received through optical communication and the captured image signal is performed. As an example, an example in which an icon image based on information received by optical communication is displayed on a captured image has been described. However, the processing using information received by optical communication and a captured image signal may be used. For example, the present invention is not limited to the example described above.

  For example, when imaging a person carrying a transmission light source, the information received by optical communication is associated with the information received by optical communication and the transmission light source on the captured image. It is also possible to notify whether a person has sent.

  FIG. 22 shows another example of the configuration of the application processor of FIG. In the example of FIG. 20, the camera unit 81, the monitor 82, the operation input unit 83, the image signal processing unit 211, the decoding processing unit 212, and the application processor 213 are common to the mobile phone 71 of FIG. However, only the configuration of the application processor 213 in FIG. 22 is different.

  That is, the application processor 213 in FIG. 22 includes an image recognition unit 341 and an information processing unit 342, and associates the pixels constituting the image sensor 122 with the pixels constituting the optical communication sensor 121, for example, in a one-to-one correspondence. Then, using the image signal from the image sensor 122 and the communication information from the optical communication sensor 121, the person carrying the transmission light source is identified from the image signal from the image sensor 122, the information of the identified person, and the optical communication A process of associating communication information from the sensor 121 is performed.

  For example, there are a plurality of persons carrying a transmission light source in the field angle boundary between the image sensor 122 and the optical communication sensor 121, and the user can select a plurality of persons carrying the transmission light source existing in the field angle boundary from the camera unit. 81 is imaged.

  Similar to the image recognition unit 331 in FIG. 20, the image recognition unit 341 captures the image signal after the signal processing from the image signal processing unit 211 and performs image recognition to thereby determine the position of the light source (pixel to be imaged). calculate. Furthermore, the image recognizing unit 341 identifies the person carrying the transmission light source by image recognition, and supplies the information of the identified person (for example, name) and the calculated position of the transmission light source to the information processing unit 342.

  The information processing unit 342 associates the pixels constituting the image sensor 122 with the pixels constituting the optical communication sensor 121 on a one-to-one basis, and calculates the received pixels from the decoding processing unit 212 and the image recognition unit 341. Based on the position of the transmission light source, the specified person information is associated with the communication information from the optical communication sensor 121. For example, the correspondence relationship such as the communication information and the transmission source that transmitted the communication information is displayed on the monitor 82. Notify through.

  Thereby, the user can grasp which communication information is transmitted to whom.

  As described above, by configuring the application processor 213, when a plurality of persons (for example, three persons) carrying the transmission light source are captured, an image is obtained using the image signal obtained by the image sensor 122. 3 persons are identified, the communication information obtained by the optical communication sensor 121 is made to correspond to the specified 3 persons, and which communication information is transmitted to the user accurately. You can also be notified.

  The present invention can also be applied to the case where the positional relationship between the optical communication sensor 121 and the image sensor 122 is inspected after the camera unit 81 is assembled.

  FIG. 23 is a diagram illustrating a configuration example of an inspection apparatus 251 that inspects the positional relationship between the optical communication sensor 121 and the image sensor 122 after the camera unit 81 is assembled. In this case, a transmission light source that transmits a blinking pattern coded with specific information is used as a light source for alignment.

  The inspection apparatus 251 is configured to be detachable from the assembled camera unit 81 to be inspected. In the example of FIG. 23, the inspection apparatus 251 is equipped with the camera unit 81 of FIG.

  The inspection device 251 includes an image signal processing unit 361, a decoding processing unit 362, an application processor 363, an operation input unit 364, and a monitor 365.

  The image signal processing unit 361 is configured in the same manner as the image signal processing unit 211 in FIG. 11, and performs color signal processing, gamma correction, and the like on the raw image signal from the image sensor 122 of the attached camera unit 81. Performs processing such as AWB and automatic exposure compensation.

  The decoding processing unit 362 is configured in the same manner as the decoding processing unit 212 of FIG. 11, and is a binary signal (that is, a light flashing detection signal) that indicates the presence or absence of light flashing detection from the optical communication sensor 121 of the attached camera unit 81. ) Is decoded.

  The application processor 363 is configured in the same manner as the application processor 213 in FIG. 11 and includes, for example, a CPU, a ROM, a RAM, and the like, and a signal corresponding to a user operation input via the operation unit 364. In addition, each part of the inspection apparatus 351 is controlled by executing various programs stored in a built-in memory.

  That is, the application processor 213 supplies a synchronization signal to the image signal processing unit 361 and the decoding processing unit 362 and executes a predetermined program, thereby internally including a light receiving pixel detection unit 371 and a pixel information acquisition unit. 372 and the defective product determination unit 373, and the pixels constituting the image sensor 122 and the pixels constituting the optical communication sensor 121 are in proportions according to the number and magnification of the optical communication sensor 121 and the image sensor 122 (current In this case, the attached camera unit 81 is inspected using the image signal from the image sensor 122 and the communication information from the optical communication sensor 121 in association with each other on a one-to-one basis, and the result is displayed on the monitor 365. To perform the process.

  Similarly to the image recognition unit 331 in FIG. 20, the light reception pixel detection unit 371 takes in the image signal after the signal processing from the image signal processing unit 361 and performs image recognition, so that the position of the transmission light source (the light reception pixel to be imaged). ) And the calculated information of the light receiving pixels is supplied to the defective product determination unit 373.

  The communication information acquisition unit 372 acquires the communication information and information on the pixel that has received the optical signal from the decode processing unit 362, and supplies the acquired communication information and information on the pixel (reception pixel) to the defective product determination unit 373.

  The defective product determination unit 373 receives, via the operation input unit 364, setting information of the camera unit 81 (resolution and angle of view of each sensor, distance d between sensors, etc.), the position (distance) where the transmission light source is installed, and its Communication information transmitted from the transmission light source and the like are stored, and from these pieces of information, pixel shift amounts of the two sensors (non-defective products) at a distance where the transmission light source is installed are calculated and stored.

  The defective product determination unit 373 reads out pixel shift amounts of the two sensors (non-defective products) at a distance where the transmission light source corresponding to the communication information from the pixel information acquisition unit 372 is installed. In addition, the defective product determination unit 373 associates the pixels configuring the image sensor 122 and the pixels configuring the optical communication sensor 121 on a one-to-one basis, and receives the light reception pixels and the pixel information acquisition unit 372 from the light reception pixel detection unit 371. Are used to calculate the positional relationship (shift amount) of these pixels.

  The defective product determination unit 373 determines whether or not the calculated pixel shift amount matches the non-defective pixel shift amount. If the pixel shift amount does not match, the camera unit 81 is determined to be a defective product. The determination result is displayed on the monitor 365 and notified to the manufacturer.

  The operation input unit 364 includes a keyboard, a mouse, and the like, and inputs an operation signal corresponding to the manufacturer's operation to the application processor 213. The monitor 365 is composed of an LCD or the like, and displays an image corresponding to the image signal from the application processor 213.

  FIG. 24 shows light receiving pixels in the optical communication sensor 121 and the image sensor 122. In the example of FIG. 24, parts corresponding to those in the example of FIG. 9 are denoted by the corresponding reference numerals, and the description thereof will be repeated, and will be omitted as appropriate.

  In the example of FIG. 24, it is assumed that the distance from the lens surface L is far from the vicinity within the field angle boundaries 131 and 132 of the optical communication sensor 121 and the image sensor 122 that are separated by the inter-sensor distance d. The projected areas of each sensor at the distances D1 and D3 are shown.

  Also in the example of FIG. 24, the dotted lines indicating the positions of the distances D1 and D3 represent the projection areas of the optical communication sensor 121 at the distances D1 and D3, and the white circled numbers shown on the dotted lines are The number from the left end of the pixel in the field angle boundary 131 of the optical communication sensor 121 is represented, and the vertical line shown above the dotted line represents the pixel projection boundary of the optical communication sensor 121.

  Similarly, the dotted lines indicating the positions of the distances D1 and D3 indicate the projection area of the image sensor 122 at the distances D1 and D3, and the black circled numbers below the dotted lines indicate the angle-of-view boundary 132 of the image sensor 122 The vertical line shown below the dotted line represents the pixel projection boundary of the image sensor 122.

  As described above with reference to FIG. 9, when the light receiving pixels of the subject (transmitting light source) on the line of sight 161 starting from the image sensor 122 are considered, the optical communication sensor 121 and the image sensor 122 are at a nearby distance D1. The transmission light source P1 is received by the first pixel from the left end of the field angle boundary 132 (black circled numeral 1) in the image sensor 122 due to the field angle deviation (that is, the inter-sensor distance d), and the optical communication sensor 121 is received. , The second pixel from the left end of the view angle boundary 131 (white circled number 2).

  At the distance D3, the image sensor 122 receives light at the first pixel from the left end of the view angle boundary 132 (black circled numeral 1), and the optical communication sensor 121 receives the first light from the left end of the view angle boundary 131. Light is received by a pixel (a white circled number 1).

  Therefore, in the inspection apparatus 351, the transmission light source P1 installed at the distance D1 from the lens surface L is received by the first pixel (black circled numeral 1) from the left end of the angle-of-view boundary 132 in the image sensor 122, and the light In the communication sensor 121, if it is confirmed that light is received by the second pixel from the left end of the angle of view boundary 131 (white circled number 2), the positional relationship between the optical communication sensor 121 and the image sensor 122 is as designed. I understand that.

  Thereby, it is possible to construct a simplified inspection process.

  If further simplification is desired, in the optical communication sensor 121 and the image sensor 122, it is confirmed that there is one pixel deviation of the pixel that receives the transmission light source P1 installed at the distance D1 from the lens surface L. It can be seen that the positional relationship between the optical communication sensor 121 and the image sensor 122 is as designed.

  Further, as shown in the light receiving pixels at the distance D1 and the distance D3, the pixel shift amount varies depending on the distance from the lens surface L of the transmission light source, so the inspection accuracy is changed according to the distance at which the transmission light source is installed. It is also possible to do.

  Furthermore, in order to improve the inspection accuracy, it is possible to install not only one transmission light source but also a plurality of transmission light sources at different positions within the field angle boundary. In this case, if each transmitting light source blinks in a different pattern, it is possible to definitely distinguish the transmitting light source.

  In the example of FIG. 24, only the H direction is shown, but the same is confirmed not only in the H direction but also in the V direction.

  Next, the inspection process of the inspection apparatus 351 in FIG. 23 will be described with reference to the flowchart in FIG. This inspection process is started when, for example, the camera unit 81 to be inspected is mounted on the inspection apparatus 351.

  When the application processor 363 starts supplying the synchronization signal (or clock signal) to the image signal processing unit 361 and the decoding processing unit 362, the image signal processing unit 361 drives the image sensor 122 from the synchronization signal. A drive signal (timing pulse) is generated, and supply of the synchronization signal and the drive signal to the image sensor 122 is started.

  Based on the synchronization signal and the drive signal from the image signal processing unit 361, the image sensor 122 outputs an image signal corresponding to the image of the subject existing within the field angle boundary 132 of the image sensor 122 via the lens 112-2. Obtaining and starting outputting the obtained image signal to the image signal processing unit 361.

  Therefore, the image signal processing unit 361 performs processing such as color signal processing, gamma correction, AWB, and automatic exposure correction on the raw image signal from the image sensor 122 in the same manner as normal signal processing of the image sensor. .

  In response to this, the light receiving pixel detection unit 371 of the application processor 363 takes in the image signal subjected to signal processing from the image signal processing unit 361 in step S71.

  In step S72, the light receiving pixel detecting unit 371 detects and detects the position of the light source (image forming light receiving pixel) by capturing the image signal after the signal processing from the image signal processing unit 361 and performing image recognition. Information on the light receiving pixels is supplied to the defective product determination unit 373.

  On the other hand, the decode processing unit 362 generates a drive signal (timing pulse) for driving the optical communication sensor 121 from the synchronization signal from the application processor 363 and starts supplying the synchronization signal and the drive signal to the optical communication sensor 121.

  Based on the synchronization signal and the drive signal from the decoding processing unit 362, the optical communication sensor 121 passes through the lens 112-1 and the subjects 2-1 and 2-2 existing in the field angle boundary 131 of the optical communication sensor 121. Communication information encoded by the blinking patterns 22-1 and 22-2 of the transmission light sources 3-1 and 3-2 included in the information is detected, and the detected information (that is, a binary signal indicating the presence / absence of light blinking detection) is detected. A certain light flashing detection signal) starts to be output to the decoding processing unit 362.

  The decode processing unit 362 performs a process of decoding a light blinking detection signal that is a binary signal indicating the presence or absence of light blinking detection from the optical communication sensor 121.

  In response to this, in step S73, the communication information acquisition unit 372 of the application processor 213 acquires the communication information obtained as a result of decoding and the information of the pixel that has received the optical signal from the decoding processing unit 362, and acquires the information. The communication information and the pixel (reception pixel) information are supplied to the defective product determination unit 373.

  In step S74, the defective product determination unit 373 reads out pixel shift amounts of the two sensors (non-defective products) at the distance at which the transmission light source corresponding to the communication information from the pixel information acquisition unit 372 is installed. The pixels constituting the sensor 122 and the pixels constituting the optical communication sensor 121 are associated one-to-one, and the light receiving pixels from the light receiving pixel detection unit 371 and the reception pixels from the pixel information acquisition unit 372 are used. The pixel positional relationship (deviation amount) is calculated.

  In step S74, the pixel of the non-defective image sensor 122 and the non-defective light corresponding to this pixel at a distance where the transmission light source corresponding to the communication information from the pixel information acquisition unit 372 is installed, not the pixel shift amount. The pixel of the communication sensor 121 is read, and in step S75, the light receiving pixel from the light receiving pixel detection unit 371 matches the pixel of the non-defective image sensor 122, and the reception pixel from the pixel information acquisition unit 372 is non-defective light. It can also be determined whether the pixel matches the pixel of the communication sensor 121.

  In this case, it is possible to perform an inspection with higher accuracy than the determination based on the deviation amount alone.

  In step S76, the defective product determination unit 373 determines whether or not the calculated pixel shift amount matches the non-defective pixel shift amount. If it is determined that they match, the process proceeds to step S77. Assume that the camera unit 81 is a non-defective product. For example, a non-defective product notification is displayed on the monitor 365 or a component mounting unit (not shown) is controlled to sort the camera unit 81 into a non-defective product line.

  If it is determined in step S76 that the calculated pixel shift amount does not match the non-defective pixel shift amount, the process determines that the camera unit 81 is defective in step S77. Notification of defective products is displayed, and a component mounting unit (not shown) is controlled to sort the camera unit 81 into defective product lines in the production line.

  As described above, an image signal from the image sensor 122 and communication information from the optical communication sensor 121 are associated with each other, and a camera to be mounted using the image signal from the image sensor 122 and the communication information from the optical communication sensor 121. Since the inspection of the part 81 is performed, a simplified inspection process can be established also in the inspection for inspecting the positional relationship between the optical communication sensor and the image sensor.

  For example, if both of the adjacent sensors are image sensors, it is easy to image a pointed subject such as a light source and check whether the pixel receiving the light source is as designed. In the case of an optical communication sensor, since it is impossible to acquire an image, it is possible to easily check the positional relationship between the optical communication sensor and the image sensor, which has been difficult.

  As a result, since the optical communication sensor and the image sensor are accurately arranged, the mobile phone 71 including the camera unit 81 that can simultaneously perform image acquisition and optical communication without degrading the image quality is Can be provided.

  In the above description, the example in which the camera unit 81 is an inspection target has been described. For example, a block in which the camera unit 81, the image signal processing unit 211, and the decoding processing unit 212 in FIG. You can also In this case, the image signal processing unit 361 and the decoding processing unit 362 are excluded from the inspection apparatus 351 in FIG.

  In the above description, the inspection process of the camera unit 81 has been described. However, a similar process can be applied to, for example, alignment of a sensor when the camera unit 81 is manufactured. In this case, for example, after performing a rough alignment with a transmission light source at a distant distance, it is possible to perform a highly accurate alignment by reducing the installation distance of the light source.

  In the above description, a mobile phone including a camera unit in which an optical communication sensor and an image sensor are arranged has been described. However, the present invention is not limited to the mobile phone, and the optical communication sensor and the image are provided. As long as it has a camera unit in which sensors are arranged, it can also be applied to PDAs (Personal Digital Assistance), personal computers, and the like.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 26 is a block diagram showing an example of the configuration of a personal computer 401 that executes the above-described series of processing by a program. A CPU (Central Processing Unit) 411 executes various processes according to a program stored in a ROM (Read Only Memory) 412 or a storage unit 418. A RAM (Random Access Memory) 413 appropriately stores programs executed by the CPU 411 and data. The CPU 411, the ROM 412, and the RAM 413 are connected to each other by a bus 414.

  An input / output interface 415 is also connected to the CPU 411 via the bus 414. The input / output interface 415 is connected to an input unit 416 including a keyboard, a mouse, and a microphone, and an output unit 417 including a display and a speaker. The CPU 411 executes various processes in response to commands input from the input unit 416. Then, the CPU 411 outputs the processing result to the output unit 417.

  The storage unit 418 connected to the input / output interface 415 includes, for example, a hard disk, and stores programs executed by the CPU 411 and various data. The communication unit 319 communicates with an external device via a network such as the Internet or a local area network.

  A program may be acquired via the communication unit 419 and stored in the storage unit 418.

  The drive 420 connected to the input / output interface 415 drives a removable medium 421 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the programs and data recorded therein. Get etc. The acquired program and data are transferred to and stored in the storage unit 418 as necessary.

  As shown in FIG. 26, a program recording medium that stores a program that is installed in a computer and is ready to be executed by the computer includes a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only). Memory, DVD (Digital Versatile Disc), a magneto-optical disk, a removable medium 421 that is a package medium composed of a semiconductor memory, a ROM 412 in which a program is temporarily or permanently stored, and a storage unit 418 It is comprised by the hard disk etc. which comprise. The program is stored in the program recording medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 419 that is an interface such as a router or a modem as necessary. Done.

  In the present specification, the step of describing the program stored in the program recording medium is not limited to the processing performed in time series in the order described, but is not necessarily performed in time series. Or the process performed separately is also included.

It is a figure explaining the principle of a two-dimensional array-like sensor. It is a figure explaining the blink pattern in which communication information was coded. It is a block diagram which shows the structural example of the sensor of FIG. It is a figure explaining the image acquisition mode and optical communication mode of the sensor of FIG. It is a figure which shows the structural example of the camera of FIG. It is a block diagram which shows the structure of one Embodiment of the mobile telephone to which this invention is applied. It is a figure which shows the structural example of the camera part of FIG. It is a figure which shows the relationship between the angle of view and optical axis of the optical communication sensor and image sensor of FIG. It is a figure explaining the light receiving pixel in the optical communication sensor and image sensor of FIG. FIG. 10 is a diagram for specifically explaining the light receiving pixels of FIG. 9. It is a block diagram which shows the example of an internal structure of the mobile telephone of FIG. It is a figure which shows the structural example of the data frame which is an optical signal from the transmission light source of FIG. 12 is a flowchart illustrating image signal processing by the application processor of FIG. 11. FIG. 12 is a block diagram illustrating another example of the internal configuration of the mobile phone in FIG. 11. It is a figure explaining the shift | offset | difference of a pixel when a 2 times optical zoom is set. FIG. 15 is a block diagram illustrating another example of the internal configuration of the mobile phone of FIG. 14. FIG. 9 is a diagram illustrating an example when only the image sensor 122 in FIG. 8 is set with a double optical zoom. FIG. 18 is a diagram for specifically explaining the light receiving pixels in FIG. 17. It is a flowchart explaining the image signal processing by the application processor of FIG. FIG. 14 is a block diagram showing still another example of the internal configuration of the mobile phone in FIG. 11. FIG. 21 is a flowchart for describing image signal processing by the application processor of FIG. 20. FIG. It is a block diagram which shows the other structural example of the application processor of FIG. It is a block diagram which shows the structure of one Embodiment of the test | inspection apparatus to which this invention is applied. It is a figure explaining the light receiving pixel in the optical communication sensor and image sensor of FIG. It is a flowchart explaining the inspection process of the inspection apparatus of FIG. It is a block diagram which shows the structural example of the personal computer to which this invention is applied.

Explanation of symbols

  71 mobile phone, 81 camera unit, 82 monitor, 83 operation input unit, 111 camera board, 121 optical communication sensor, 122 image sensor, 211 image signal processing unit, 212 decode processing unit, 213 application processor, 231 image superimposing unit, 232 Image generation unit, 251-1, 251-2 zoom drive unit, 261 zoom control signal output unit, 301 corresponding pixel conversion unit, 331 image recognition unit, 332 deviation correction amount calculation unit, 341 image recognition unit, 342 information processing unit, 351 inspection device, 361 image signal processing unit, 362 decoding processing unit, 363 application processor, 364 operation input unit, 365 monitor

Claims (10)

In an imaging device for imaging a subject,
An image sensor for inputting an image signal corresponding to a subject provided with a transmission light source for transmitting an optical signal encoded with information;
An optical communication sensor that is arranged with a substantially angle of view at a position close to the image sensor, and that receives the optical signal transmitted by the transmission light source; and
Information obtained by associating the pixels constituting the image sensor with the pixels constituting the optical communication sensor and decoding the image signal input by the image sensor and the optical signal received by the optical communication sensor An image pickup apparatus comprising: signal processing means for performing processing using a signal. The imaging apparatus according to claim 1, wherein the information is information related to the subject. Further comprising display means for displaying an image corresponding to the image signal,
The signal processing means includes
Image generating means for generating an image corresponding to information obtained by decoding the optical signal;
The pixels constituting the image sensor and the pixels constituting the optical communication sensor are associated with each other, and the image signal of the image generated by the image generation unit is superimposed on the image signal input by the image sensor. Image superimposing means for performing processing to perform,
The imaging apparatus according to claim 2, wherein the display unit displays an image corresponding to the image signal superimposed by the signal processing unit. The signal processing means includes
Image recognition means for recognizing a subject from an image corresponding to the image signal input by the image sensor;
A process of setting a subject recognized by the image recognition unit as a source of information obtained by decoding the optical signal by associating a pixel constituting the image sensor with a pixel constituting the optical communication sensor The imaging apparatus according to claim 1, further comprising a transmission source setting unit. First optical system driving means for driving the optical system of the image sensor;
Second optical system driving means for driving the optical system of the optical communication sensor;
Zoom control means for controlling the first and second optical system driving means so as to interlock the optical system of the image sensor and the optical system of the optical communication sensor based on a zoom value corresponding to a user operation. The imaging device according to claim 1. Optical system driving means for driving the optical system of the image sensor;
Zoom control means for controlling driving of the optical system by the optical system driving means based on a zoom value corresponding to a user operation,
The signal processing means includes
Corresponding pixel calculation means for calculating a corresponding pixel of the optical communication sensor corresponding to a pixel of the image sensor based on a zoom value corresponding to a user operation,
Based on the corresponding pixel of the optical communication sensor calculated by the corresponding pixel calculating means, the pixel configuring the image sensor and the pixel configuring the optical communication sensor are associated with each other and input by the image sensor The imaging device according to claim 1, wherein processing is performed using information obtained by decoding the image signal and the optical signal received by the optical communication sensor. The signal processing means includes
Pixel detection means for detecting a pixel corresponding to the transmission light source from an image corresponding to the image signal input by the image sensor;
A correction amount calculation unit that calculates a pixel shift correction amount based on the pixel corresponding to the transmission light source detected by the pixel detection unit and the pixel of the optical communication sensor that has received the optical signal;
Based on the correction amount of the pixel shift calculated by the correction amount calculation means, the pixels constituting the image sensor and the pixels constituting the optical communication sensor are associated with each other and input by the image sensor. The imaging apparatus according to claim 1, wherein processing is performed using information obtained by decoding an image signal and the optical signal received by the optical communication sensor. In an image processing method of an imaging apparatus that images a subject,
Pixels constituting an image sensor that inputs an image signal corresponding to a subject including a light source that transmits an optical signal in which information is encoded;
The image sensor is disposed in a position close to the image sensor with a substantially angle of view, and is associated with a pixel constituting the optical communication sensor that receives the optical signal transmitted from the transmission light source, and is input by the image sensor. An image processing method including a step of performing processing using the image signal and information obtained by decoding the optical signal received by the optical communication sensor. A program that causes an imaging device that images a subject to perform image processing,
Pixels constituting an image sensor that inputs an image signal corresponding to a subject including a light source that transmits an optical signal in which information is encoded;
The image sensor is disposed in a position close to the image sensor with a substantially angle of view, and is associated with a pixel constituting the optical communication sensor that receives the optical signal transmitted from the transmission light source, and is input by the image sensor. A program including a step of performing processing using the image signal and information obtained by decoding the optical signal received by the optical communication sensor.   A recording medium on which the program according to claim 9 is recorded.

Images