US20230074088A1 - Photographing device capable of outputting tagged image frame - Google Patents
Photographing device capable of outputting tagged image frame Download PDFInfo
- Publication number
- US20230074088A1 US20230074088A1 US17/988,059 US202217988059A US2023074088A1 US 20230074088 A1 US20230074088 A1 US 20230074088A1 US 202217988059 A US202217988059 A US 202217988059A US 2023074088 A1 US2023074088 A1 US 2023074088A1
- Authority
- US
- United States
- Prior art keywords
- image
- photographing device
- image frame
- output
- processor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000605 extraction Methods 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 8
- 230000008859 change Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000000034 method Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
- H04N5/765—Interface circuits between an apparatus for recording and another apparatus
- H04N5/77—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera
- H04N5/772—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera the recording apparatus and the television camera being placed in the same enclosure
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G11B27/34—Indicating arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
-
- H04N5/2353—
Definitions
- This disclosure generally relates to a photographing system and, more particularly, to a smart photographing system that utilizes a photographing device having two output interfaces to output different image frames respectively to a processor and a back end to perform the image feature recognition and tagging as well as the image recording.
- FIG. 1 it is a block diagram of a conventional video system including an image sensor 11 and a back end circuit 13 .
- the image sensor 11 is used to monitor an environmental change and output a video compatible with the Full HD or higher resolution format to the back end circuit 13 .
- the back end circuit 13 records the video and then performs the image analysis to tag image features in the recorded video.
- the back end circuit 13 has higher power consumption.
- the power saving is an important issue such that the total power consumption of a system should be reduced as much as possible.
- the present disclosure provides a smart photographing system that reduces the total power consumption by reducing data amount processed by a back end circuit thereof.
- the present disclosure provides a smart photographing system that adopts a photographing device having two output interfaces to respectively output image frames of different resolutions to different receiving ends to finish the feature tagging before the image recording.
- the present disclosure provides a photographing device including an image sensor and a processor.
- the image sensor is configured to capture a series of image data.
- the processor is configured to receive a first image frame, which corresponds to a first part of the series of image data and has a first size, control the photographing device to output a second image frame, which corresponds to a second part of the series of image data and has a second size, and a tag associated with a predetermined feature to downstream of the photographing device upon identifying the first image frame containing the predetermined feature, and control the photographing device to continuously output successive second image frames for a predetermined interval.
- the present disclosure further provides a photographing device including an image sensor and a processor.
- the image sensor is configured to capture a series of image data.
- the processor is configured to receive a first image frame, which corresponds to a first part of the series of image data and has a first size, and control the photographing device to output a second image frame, which corresponds to a second part of the series of image data and has a second size, and a tag associated with a predetermined feature to downstream of the photographing device upon identifying the first image frame containing the predetermined feature, wherein before the processor identifies the first image frame containing the predetermined feature, the photographing device is configured not to output any second image frame.
- the present disclosure further provides a photographing device including an image sensor, a first output interface, a second output interface and a processor.
- the image sensor is configured to capture image data of multiple pixels.
- the first output interface is coupled to the image sensor, and configured to output a first image frame, which corresponds to a first part of the series of image data and has a first size.
- the second output interface is coupled to the image sensor, and configured to output a second image frame, which corresponds to a second part of the series of image data and has a second size, to downstream of the photographing device.
- the processor configured to receive the first image frame, upon identifying the first image frame containing a predetermined feature, control the image sensor to output the second image frame via the second output interface for a predetermined interval and to output at least one first image frame via the first output interface within the predetermined interval, and extend the predetermined interval when the predetermined feature or a new predetermined feature is identified in the at least one first image frame within the predetermined interval.
- the feature tag of the present disclosure is referred to any tags instead of a time tag such as the moving object tag, ID tag, face tag, skin color tag, human shape tag, vehicle tag, license plate tag and so on.
- the tag is additional information added to pixel data of the second image frame.
- FIG. 1 is a block diagram of a conventional video system.
- FIG. 2 is a block diagram of a photographing system according to one embodiment of the present disclosure.
- FIG. 3 is an operational schematic diagram of a photographing device according to one embodiment of the present disclosure.
- FIG. 4 is a block diagram of a photographing system according to another embodiment of the present disclosure.
- the present disclosure is applicable to an image processing system that transmits captured image frames to a back end circuit for post-processing.
- the image processing system is, for example, a security monitoring system.
- the back end is arranged to record a plurality of images (or referred to a video) outputted by a photographing device, and a video interval desired to be watched in playing the video on a screen is selected by selecting the recorded feature tag to realize a smart photographing system.
- FIG. 2 it is a block diagram of a smart photographing system 200 according to one embodiment of the present disclosure, including a photographing device 20 and a back end circuit 9 coupled to each other, wherein the back end circuit 9 has the function of image recording (e.g., in a memory) and playing (e.g., via a screen).
- the back end circuit 9 is, for example, a computer system such as a notebook computer, a tablet computer, a desktop computer or a central monitoring system. According to different applications, the back end circuit 9 has different play modes such as fast-forward, backward and selecting video time interval.
- the photographing system 200 records environmental sound, and the back end circuit 9 has the function of playing audio data.
- the photographing device 20 and the back end circuit 9 are arranged as a single device, or arranged as two separated devices coupled to each other in a wired or wireless manner without particular limitations.
- the back end circuit 9 is, for example, in a remote central server out of the photographing device 20 .
- the photographing device 20 is, for example, a sensor chip formed as an IC package, and has pins for communicating with external electronic devices.
- the photographing device 20 includes an image sensor 21 , a first output interface 22 , a second output interface 23 and a processor 24 .
- the first output interface 22 is coupled to the processor 24 , and outputs first image frames Im 1 having a first size to the processor 24 for the image recognition and analysis.
- the second output interface 23 is coupled to a back end circuit 9 out of the photographing device 20 via the pin (not shown in figure), or by a wired or wireless connection, and outputs second image frames Im 2 having a second size to the back end circuit 9 , e.g., via a signal transmission line, a bus line and/or wireless channel
- the first size is preferably much smaller than the second size.
- the second size is compatible with the full HD format or higher formats for recording the video suitable to be watched by users; and the first size is compatible with the SD format or lower formats to reduce data amount processed by the processor 24 .
- the image sensor 21 is, for example, a CCD image sensor, a CMOS image sensor or other optical sensors for converting light energy to electrical signals.
- the image sensor 21 includes a plurality of pixels for generating image data within every frame period to the first output interface 22 or the second output interface 23 .
- the image sensor 21 includes a pixel array for generating the image data, and has a sampling circuit (e.g., CDS circuit) for sampling the image data from every pixel.
- the sampled image data is then converted into digital data by an analog to digital converter (ADC) to form the first image frame Im 1 or the second image frame Im 2 .
- ADC analog to digital converter
- the image sensor 21 captures a series of image data, corresponding to successive image frame, at a predetermined frame rate.
- the first image frame corresponds to a first part of the series of image data
- the second image frame corresponds to a second part of the series of image data.
- the first and second parts of the series of image data correspond to image data of a same image frame or different image frames.
- the first image frame Im 1 is acquired within the frame period by turning off a part of pixels of the pixel array of the image sensor 21 , i.e. the first image frame Im 1 containing the image data outputted by a part of pixels of the pixel array.
- the first image frame is generated by downsampling the image data outputted by the image sensor 21 , but not limited thereto. Other techniques suitable to reduce the size of image frames outputted by an image sensor are also applicable to the present disclosure.
- the processor 24 is, for example, an application specific integrated circuit (ASIC) or a digital signal processor (DSP), and used to receive a first image frame Im 1 to identify whether the first image frame Im 1 includes a predetermined feature. For example, when the first image frame Im 1 contains a moving object (e.g., by comparing multiple image frames), the first image frame Im 1 is identified to contain the predetermined feature, but not limited to.
- the processor 24 identifies a face, a human shape, a predetermined identification (ID), a predetermined vehicle, a predetermined license plate, skin color and so on (e.g., using the machine learning or comparing with pre-stored features) to indicate that the first image frame Im 1 contains the predetermined feature.
- the processor 24 informs the image sensor 21 to output successive image frames (or video), i.e. the second image Im 2 herein, to the back end circuit 9 for the image recording.
- FIG. 3 it is a schematic diagram of several operational aspects of the photographing device 20 according to some embodiments of the present disclosure.
- each arrow symbol indicates one image frame.
- the first row in FIG. 3 indicates image frames generated by the image sensor 21 , and each arrow symbol in FIG. 3 indicates image data of one image frame is captured.
- the processor 24 controls the image sensor 21 to continuously (i.e. not outputting the first image frame Im 1 ) output second image frames Im 2 for a predetermined interval (e.g., a time interval between T 1 and T 2 ), and adds a tag associated with the predetermined feature on every second image frame Im 2 outputted within the predetermined interval.
- a predetermined interval e.g., a time interval between T 1 and T 2
- the tag is included, for example, within the data header of the every second image frame Im 2 , e.g., showing by the region filled with slant lines in FIG. 2 .
- the tag may be different corresponding to different image features.
- the tag contains at least one of a moving object tag, an ID tag, a face tag, a skin color tag, a human shape tag, a vehicle tag and a license plate tag, but not limited thereto.
- the processor 24 adds one or more than one tags to the second image frame Im 2 according to different predetermined features using, for example, a register 25 to change a digital value, wherein the processor 24 is arranged to tag predetermined types of different features, and a number of said types is determined according to different applications and the processing ability of the processor 24 .
- the image sensor 21 does not output any second image frame Im 2 to the back end circuit 9 via the second output interface 23 .
- the processor 24 identifies that the first image frame Im 1 contains the predetermined feature, it means that the photographed environment has information desired to be recorded such that a recording mode (e.g., between T 1 and T 2 ) is entered.
- a recording mode e.g., between T 1 and T 2
- the back end circuit 9 records both image data and tagged data of the second image frame Im 2 within the predetermined interval between T 1 and T 2 , the image sensor 21 does not output the first image frame Im 1 via the first output interface 22 .
- the processor 24 is shut down or enters a sleep mode in the recording mode.
- the image sensor 21 further receives an auto exposure control signal AE 2 from the back end circuit 9 , wherein AE 2 is generated by a processor (e.g., a CPU or MCU) of the back end circuit 9 by identifying, for example, brightness of the second image frame Im 2 Meanwhile, as the processor 24 is in sleeping or shut down status, the processor 24 does not output an auto exposure control signal AE 1 (e.g., generated by the processor 24 by identifying brightness of the first image frame Im 1 ) to the image sensor 21 .
- the auto exposure control signal AE 1 is sent to the image sensor 21 before the recording mode is entered.
- the image sensor 21 When the predetermined interval is over at T 2 , the image sensor 21 outputs (e.g., automatically or controlled by the processor 24 ) the first image frame Im 1 (e.g., image frame at time T 3 ) to the processor 24 via the first output interface 22 again.
- the processor 24 identifies whether the first image frames Im 1 after time T 3 (including T 3 ) contain the predetermined feature or not, and stops outputting the second image frame Im 2 to downstream of the photographing device 20 via the second output interface 23 .
- the processor 24 further identifies one first image frame Im 1 after time T 3 contains the predetermined feature, the recording mode is entered again; and since the operation from recognizing the predetermined feature and entering the recording mode have been illustrated above, details thereof are not repeated herein.
- the first output interface 22 outputs the first image frame Im 1 to the processor 24 from time to time (predetermined) within the predetermined interval T 0 -T 2 . If the processor 24 continuously identifies the predetermined feature or another new predetermined feature within the predetermined interval T 0 -T 2 , the processor 24 automatically extends the predetermined interval T 0 -T 2 . More specifically, the predetermined interval T 0 -T 2 is extendable depending on whether any predetermined feature exists in the first image frame Im 1 within the predetermined interval T 0 -T 2 .
- the processor 24 controls the image sensor 21 to alternatively output a second image frame Im 2 (e.g., image frame at time T 1 ) via the second output interface 23 and output a first image frame Im 1 via the first output interface 22 , and adds at least one tag, which is illustrated above and thus details thereof are not repeated herein, associated with the predetermined feature to the second image frame Im.
- a second image frame Im 2 e.g., image frame at time T 1
- the processor 24 controls the image sensor 21 to alternatively output a second image frame Im 2 (e.g., image frame at time T 1 ) via the second output interface 23 and output a first image frame Im 1 via the first output interface 22 , and adds at least one tag, which is illustrated above and thus details thereof are not repeated herein, associated with the predetermined feature to the second image frame Im.
- the image sensor 21 does not output any second image frame Im 2 to downstream of the photographing device 20 via the second output interface 23 .
- the processor 24 After entering a recording mode (e.g., time interval between T 1 and T 2 ), the processor 24 receives the first image frame Im 1 with a lower frequency (e.g., a half shown in FIG. 3 , but not limited thereto), and identifies whether every received first image frame Im 1 contains a predetermined feature, but the frame rate of the image sensor 21 is not changed. That is, when identifying that any first image frame Im 1 contains the predetermined feature, the processor 24 controls the image sensor 21 to output at least one (e.g., one being shown in FIG.
- the processor 24 controls the image sensor 21 to output the first image frame Im 1 via the first output interface 22 but not output the second image frame Im 2 via the second output interface 23 .
- the image sensor 21 performs the auto exposure according to the auto exposure control signal AE 1 from the processor 24 or according to the auto exposure control signal AE 2 from the back end circuit 9 without particular limitations.
- the image sensor 21 does not output image frames via the first output interface 22 and the second output interface 23 simultaneously.
- the photographing system 200 just continuously identifies the predetermined feature in the first image frames Im 1 but does not record images, e.g., the back end circuit 9 being turned off.
- second image frames Im 2 are outputted continuously or separated by at least one first image frame Im 1 for the back end circuit 9 to the image recording as shown in FIG. 3 .
- the first output interface 22 and the second output interface 23 output a first image frame Im 1 and a second image frame Im 2 in parallel, e.g., the first image frame Im 1 and the second image frame Im 2 being retrieved from the image data of the same image frame.
- the processor 24 identifies whether the first image frame Im 1 contains a predetermined image feature. If the first image frame Im 1 is identified containing the predetermined feature, the second output interface 23 outputs the second image frame Im 2 with at least one tag. On the contrary, if the first image frame Im 1 is identified not containing the predetermined feature, the second output interface 23 does not outputs the second image frame Im 2 out of the photographing device 200 .
- the smart photographing system 200 of the present disclosure further includes a passive infrared radiation (PIR) sensor.
- the processor 24 identifies whether to output the second image frame Im 2 via the second output interface 23 to the back end circuit 9 for the image recording according to output results of both the PIR sensor and the image sensor 21 (e.g., one of them detecting a moving object or human body).
- the operation is similar to the above embodiments only the processor 24 further receiving the detected result from the PIR sensor to accordingly identify a human body, and thus details thereof are not illustrated herein.
- FIG. 4 it is a schematic diagram of a photographing device 400 according to another embodiment of the present disclosure.
- the photographing device 400 includes one output interface 43 for outputting an image frame to both the downstream circuit and the processor 44 .
- the processor 44 identifies whether the image frame Im contains a predetermined feature. If the image frame Im is identified containing the predetermined feature, the output interface 43 outputs the image frame with at least one tag associated with the predetermined feature to the back end circuit 9 ; whereas, if the mage frame Im is identified not containing the predetermined feature, the output interface 43 does not output the image frame Im to the back end circuit 9 . That is, the output of the image frame Im to the back end circuit 9 waits for the identifying process performed by the processor 24 .
- an auto exposure control signal is used to control, for example, an exposure interval of the image sensor 21 , light source intensity and a gain value to change average brightness of the image frame generated by the image sensor 21 to be within a suitable range.
- the tag indicates a simple analyzed result of the first image frame Im 1 , e.g., indicating the first image frame Im 1 containing a face, human skin color, a human shape object or a vehicle.
- the processor of the back end circuit 9 has stronger calculation ability, and said processor performs the operation requiring more calculation such as performing the ID recognition or license plate recognition according to the second image frame Im 2 .
- a back end circuit performs both the image recording and the feature tagging, and the image sensor outputs image frames having only one size to the back end circuit for the image recording.
- the present disclosure further provides a photographing device generating image frames of two sizes (e.g. referring to FIG. 2 ) that recognizes a triggering object in a low resolution image frame at first and then outputs a tagged high resolution image frame to an external back end circuit for the image recording.
- the back end circuit needs not to perform the feature tag anymore.
Abstract
Description
- The present application is a continuation application of U.S. patent application Ser. No. 17/515,575, filed on Nov. 1, 2021, which is a continuation application of U.S. patent application Ser. No. 16/924,285, filed on Jul. 9, 2020, which is a continuation application of U.S. patent application Ser. No. 16/431,788, filed on Jun. 5, 2019, the disclosures of which are hereby incorporated by reference herein in their entirety.
- To the extent any amendments, characterizations, or other assertions previously made (in this or in any related patent applications or patents, including any parent, sibling, or child) with respect to any art, prior or otherwise, could be construed as a disclaimer of any subject matter supported by the present disclosure of this application, Applicant hereby rescinds and retracts such disclaimer. Applicant also respectfully submits that any prior art previously considered in any related patent applications or patents, including any parent, sibling, or child, may need to be re-visited.
- This disclosure generally relates to a photographing system and, more particularly, to a smart photographing system that utilizes a photographing device having two output interfaces to output different image frames respectively to a processor and a back end to perform the image feature recognition and tagging as well as the image recording.
- Referring to
FIG. 1 , it is a block diagram of a conventional video system including animage sensor 11 and aback end circuit 13. Theimage sensor 11 is used to monitor an environmental change and output a video compatible with the Full HD or higher resolution format to theback end circuit 13. Theback end circuit 13 records the video and then performs the image analysis to tag image features in the recorded video. - Generally, the
back end circuit 13 has higher power consumption. Nowadays, the power saving is an important issue such that the total power consumption of a system should be reduced as much as possible. - Accordingly, the present disclosure provides a smart photographing system that reduces the total power consumption by reducing data amount processed by a back end circuit thereof.
- The present disclosure provides a smart photographing system that adopts a photographing device having two output interfaces to respectively output image frames of different resolutions to different receiving ends to finish the feature tagging before the image recording.
- The present disclosure provides a photographing device including an image sensor and a processor. The image sensor is configured to capture a series of image data. The processor is configured to receive a first image frame, which corresponds to a first part of the series of image data and has a first size, control the photographing device to output a second image frame, which corresponds to a second part of the series of image data and has a second size, and a tag associated with a predetermined feature to downstream of the photographing device upon identifying the first image frame containing the predetermined feature, and control the photographing device to continuously output successive second image frames for a predetermined interval.
- The present disclosure further provides a photographing device including an image sensor and a processor. The image sensor is configured to capture a series of image data. The processor is configured to receive a first image frame, which corresponds to a first part of the series of image data and has a first size, and control the photographing device to output a second image frame, which corresponds to a second part of the series of image data and has a second size, and a tag associated with a predetermined feature to downstream of the photographing device upon identifying the first image frame containing the predetermined feature, wherein before the processor identifies the first image frame containing the predetermined feature, the photographing device is configured not to output any second image frame.
- The present disclosure further provides a photographing device including an image sensor, a first output interface, a second output interface and a processor. The image sensor is configured to capture image data of multiple pixels. The first output interface is coupled to the image sensor, and configured to output a first image frame, which corresponds to a first part of the series of image data and has a first size. The second output interface is coupled to the image sensor, and configured to output a second image frame, which corresponds to a second part of the series of image data and has a second size, to downstream of the photographing device. The processor configured to receive the first image frame, upon identifying the first image frame containing a predetermined feature, control the image sensor to output the second image frame via the second output interface for a predetermined interval and to output at least one first image frame via the first output interface within the predetermined interval, and extend the predetermined interval when the predetermined feature or a new predetermined feature is identified in the at least one first image frame within the predetermined interval.
- The feature tag of the present disclosure is referred to any tags instead of a time tag such as the moving object tag, ID tag, face tag, skin color tag, human shape tag, vehicle tag, license plate tag and so on. The tag is additional information added to pixel data of the second image frame.
- Other objects, advantages, and novel features of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.
-
FIG. 1 is a block diagram of a conventional video system. -
FIG. 2 is a block diagram of a photographing system according to one embodiment of the present disclosure. -
FIG. 3 is an operational schematic diagram of a photographing device according to one embodiment of the present disclosure. -
FIG. 4 is a block diagram of a photographing system according to another embodiment of the present disclosure. - It should be noted that, wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.
- The present disclosure is applicable to an image processing system that transmits captured image frames to a back end circuit for post-processing. The image processing system is, for example, a security monitoring system.
- One objective of the present disclosure is to reduce loading of the backed thereby reducing total power consumption of the system. The back end is arranged to record a plurality of images (or referred to a video) outputted by a photographing device, and a video interval desired to be watched in playing the video on a screen is selected by selecting the recorded feature tag to realize a smart photographing system.
- Referring to
FIG. 2 , it is a block diagram of asmart photographing system 200 according to one embodiment of the present disclosure, including aphotographing device 20 and aback end circuit 9 coupled to each other, wherein theback end circuit 9 has the function of image recording (e.g., in a memory) and playing (e.g., via a screen). Theback end circuit 9 is, for example, a computer system such as a notebook computer, a tablet computer, a desktop computer or a central monitoring system. According to different applications, theback end circuit 9 has different play modes such as fast-forward, backward and selecting video time interval. In some embodiments, thephotographing system 200 records environmental sound, and theback end circuit 9 has the function of playing audio data. - The photographing
device 20 and theback end circuit 9 are arranged as a single device, or arranged as two separated devices coupled to each other in a wired or wireless manner without particular limitations. Theback end circuit 9 is, for example, in a remote central server out of the photographingdevice 20. - The photographing
device 20 is, for example, a sensor chip formed as an IC package, and has pins for communicating with external electronic devices. - The
photographing device 20 includes animage sensor 21, afirst output interface 22, asecond output interface 23 and aprocessor 24. - The
first output interface 22 is coupled to theprocessor 24, and outputs first image frames Im1 having a first size to theprocessor 24 for the image recognition and analysis. Thesecond output interface 23 is coupled to aback end circuit 9 out of thephotographing device 20 via the pin (not shown in figure), or by a wired or wireless connection, and outputs second image frames Im2 having a second size to theback end circuit 9, e.g., via a signal transmission line, a bus line and/or wireless channel - In one non-limiting embodiment, the first size is preferably much smaller than the second size. For example, the second size is compatible with the full HD format or higher formats for recording the video suitable to be watched by users; and the first size is compatible with the SD format or lower formats to reduce data amount processed by the
processor 24. - The
image sensor 21 is, for example, a CCD image sensor, a CMOS image sensor or other optical sensors for converting light energy to electrical signals. Theimage sensor 21 includes a plurality of pixels for generating image data within every frame period to thefirst output interface 22 or thesecond output interface 23. For example, theimage sensor 21 includes a pixel array for generating the image data, and has a sampling circuit (e.g., CDS circuit) for sampling the image data from every pixel. The sampled image data is then converted into digital data by an analog to digital converter (ADC) to form the first image frame Im1 or the second image frame Im2. - The
image sensor 21 captures a series of image data, corresponding to successive image frame, at a predetermined frame rate. The first image frame corresponds to a first part of the series of image data, and the second image frame corresponds to a second part of the series of image data. The first and second parts of the series of image data correspond to image data of a same image frame or different image frames. - To cause the first image frame Im1 to be smaller than the second image frame Im2, in one aspect the first image frame Im1 is acquired within the frame period by turning off a part of pixels of the pixel array of the
image sensor 21, i.e. the first image frame Im1 containing the image data outputted by a part of pixels of the pixel array. In another aspect, the first image frame is generated by downsampling the image data outputted by theimage sensor 21, but not limited thereto. Other techniques suitable to reduce the size of image frames outputted by an image sensor are also applicable to the present disclosure. - The
processor 24 is, for example, an application specific integrated circuit (ASIC) or a digital signal processor (DSP), and used to receive a first image frame Im1 to identify whether the first image frame Im1 includes a predetermined feature. For example, when the first image frame Im1 contains a moving object (e.g., by comparing multiple image frames), the first image frame Im1 is identified to contain the predetermined feature, but not limited to. Theprocessor 24 identifies a face, a human shape, a predetermined identification (ID), a predetermined vehicle, a predetermined license plate, skin color and so on (e.g., using the machine learning or comparing with pre-stored features) to indicate that the first image frame Im1 contains the predetermined feature. When the first image frame Im contains the predetermined feature, theprocessor 24 informs theimage sensor 21 to output successive image frames (or video), i.e. the second image Im2 herein, to theback end circuit 9 for the image recording. - Referring to
FIG. 3 , it is a schematic diagram of several operational aspects of the photographingdevice 20 according to some embodiments of the present disclosure. InFIG. 3 , each arrow symbol indicates one image frame. The first row inFIG. 3 indicates image frames generated by theimage sensor 21, and each arrow symbol inFIG. 3 indicates image data of one image frame is captured. - In an aspect I, when identifying that the first image frame Im1 (e.g., image frame at time T0) contains a predetermined feature, the
processor 24 controls theimage sensor 21 to continuously (i.e. not outputting the first image frame Im1) output second image frames Im2 for a predetermined interval (e.g., a time interval between T1 and T2), and adds a tag associated with the predetermined feature on every second image frame Im2 outputted within the predetermined interval. - The tag is included, for example, within the data header of the every second image frame Im2, e.g., showing by the region filled with slant lines in FIG. 2. The tag may be different corresponding to different image features. For example, the tag contains at least one of a moving object tag, an ID tag, a face tag, a skin color tag, a human shape tag, a vehicle tag and a license plate tag, but not limited thereto. The
processor 24 adds one or more than one tags to the second image frame Im2 according to different predetermined features using, for example, aregister 25 to change a digital value, wherein theprocessor 24 is arranged to tag predetermined types of different features, and a number of said types is determined according to different applications and the processing ability of theprocessor 24. - More specifically in the aspect I, before the
processor 24 identifies that the first image frame Im1 contains the predetermined feature, theimage sensor 21 does not output any second image frame Im2 to theback end circuit 9 via thesecond output interface 23. When theprocessor 24 identifies that the first image frame Im1 contains the predetermined feature, it means that the photographed environment has information desired to be recorded such that a recording mode (e.g., between T1 and T2) is entered. In the recording mode, theback end circuit 9 records both image data and tagged data of the second image frame Im2 Within the predetermined interval between T1 and T2, theimage sensor 21 does not output the first image frame Im1 via thefirst output interface 22. To further reduce the power consumption, theprocessor 24 is shut down or enters a sleep mode in the recording mode. - Within the predetermined interval between T1 and T2, to normally perform an auto exposure operation, the
image sensor 21 further receives an auto exposure control signal AE2 from theback end circuit 9, wherein AE2 is generated by a processor (e.g., a CPU or MCU) of theback end circuit 9 by identifying, for example, brightness of the second image frame Im2 Meanwhile, as theprocessor 24 is in sleeping or shut down status, theprocessor 24 does not output an auto exposure control signal AE1 (e.g., generated by theprocessor 24 by identifying brightness of the first image frame Im1) to theimage sensor 21. The auto exposure control signal AE1 is sent to theimage sensor 21 before the recording mode is entered. - When the predetermined interval is over at T2, the
image sensor 21 outputs (e.g., automatically or controlled by the processor 24) the first image frame Im1 (e.g., image frame at time T3) to theprocessor 24 via thefirst output interface 22 again. Theprocessor 24 identifies whether the first image frames Im1 after time T3 (including T3) contain the predetermined feature or not, and stops outputting the second image frame Im2 to downstream of the photographingdevice 20 via thesecond output interface 23. When theprocessor 24 further identifies one first image frame Im1 after time T3 contains the predetermined feature, the recording mode is entered again; and since the operation from recognizing the predetermined feature and entering the recording mode have been illustrated above, details thereof are not repeated herein. - In a non-limiting aspect, the
first output interface 22 outputs the first image frame Im1 to theprocessor 24 from time to time (predetermined) within the predetermined interval T0-T2. If theprocessor 24 continuously identifies the predetermined feature or another new predetermined feature within the predetermined interval T0-T2, theprocessor 24 automatically extends the predetermined interval T0-T2. More specifically, the predetermined interval T0-T2 is extendable depending on whether any predetermined feature exists in the first image frame Im1 within the predetermined interval T0-T2. - In an aspect II, when identifying that the first image frame Im1 (e.g., image frame at time T0) contains a predetermined feature, the
processor 24 controls theimage sensor 21 to alternatively output a second image frame Im2 (e.g., image frame at time T1) via thesecond output interface 23 and output a first image frame Im1 via thefirst output interface 22, and adds at least one tag, which is illustrated above and thus details thereof are not repeated herein, associated with the predetermined feature to the second image frame Im. - More specifically in the aspect II, before the
processor 24 identifies that the first image frame Im1 contains the predetermined feature, theimage sensor 21 does not output any second image frame Im2 to downstream of the photographingdevice 20 via thesecond output interface 23. After entering a recording mode (e.g., time interval between T1 and T2), theprocessor 24 receives the first image frame Im1 with a lower frequency (e.g., a half shown inFIG. 3 , but not limited thereto), and identifies whether every received first image frame Im1 contains a predetermined feature, but the frame rate of theimage sensor 21 is not changed. That is, when identifying that any first image frame Im1 contains the predetermined feature, theprocessor 24 controls theimage sensor 21 to output at least one (e.g., one being shown inFIG. 3 , but not limited to) second image frame Im2 via thesecond output interface 23 to theback end circuit 9 and tags the outputted second image frame Im2, wherein the tag is determined according to a first image frame Im1 prior to the outputted second image frame Im. When identifying that the predetermined feature disappears from the first image frame Im1 (e.g., image frame at time T3), theprocessor 24 controls theimage sensor 21 to output the first image frame Im1 via thefirst output interface 22 but not output the second image frame Im2 via thesecond output interface 23. - In the aspect II, within the recording mode (e.g., between T1 and T2), as the
processor 24 is continuously in operation, theimage sensor 21 performs the auto exposure according to the auto exposure control signal AE1 from theprocessor 24 or according to the auto exposure control signal AE2 from theback end circuit 9 without particular limitations. - More specifically, in the first and second aspects, as the first image frame Im1 and the second image frame Im2 are used for different purposes, the
image sensor 21 does not output image frames via thefirst output interface 22 and thesecond output interface 23 simultaneously. When the first image frame Im1 does not contain a predetermined feature, the photographingsystem 200 just continuously identifies the predetermined feature in the first image frames Im1 but does not record images, e.g., theback end circuit 9 being turned off. When the first image frame Im1 contains the predetermined feature, second image frames Im2 are outputted continuously or separated by at least one first image frame Im1 for theback end circuit 9 to the image recording as shown inFIG. 3 . - However in an aspect III, the
first output interface 22 and thesecond output interface 23 output a first image frame Im1 and a second image frame Im2 in parallel, e.g., the first image frame Im1 and the second image frame Im2 being retrieved from the image data of the same image frame. Theprocessor 24 identifies whether the first image frame Im1 contains a predetermined image feature. If the first image frame Im1 is identified containing the predetermined feature, thesecond output interface 23 outputs the second image frame Im2 with at least one tag. On the contrary, if the first image frame Im1 is identified not containing the predetermined feature, thesecond output interface 23 does not outputs the second image frame Im2 out of the photographingdevice 200. - In some embodiments, the smart photographing
system 200 of the present disclosure further includes a passive infrared radiation (PIR) sensor. In this case, theprocessor 24 identifies whether to output the second image frame Im2 via thesecond output interface 23 to theback end circuit 9 for the image recording according to output results of both the PIR sensor and the image sensor 21 (e.g., one of them detecting a moving object or human body). The operation is similar to the above embodiments only theprocessor 24 further receiving the detected result from the PIR sensor to accordingly identify a human body, and thus details thereof are not illustrated herein. - Referring to
FIG. 4 , it is a schematic diagram of a photographingdevice 400 according to another embodiment of the present disclosure. The photographingdevice 400 includes oneoutput interface 43 for outputting an image frame to both the downstream circuit and theprocessor 44. Theprocessor 44 identifies whether the image frame Im contains a predetermined feature. If the image frame Im is identified containing the predetermined feature, theoutput interface 43 outputs the image frame with at least one tag associated with the predetermined feature to theback end circuit 9; whereas, if the mage frame Im is identified not containing the predetermined feature, theoutput interface 43 does not output the image frame Im to theback end circuit 9. That is, the output of the image frame Im to theback end circuit 9 waits for the identifying process performed by theprocessor 24. - The operation of this embodiment also implemented using
FIG. 3 , e.g., Im1 shown inFIG. 3 is replaced by Im2 More specifically, the difference betweenFIG. 4 andFIG. 2 is that inFIG. 4 , asingle output interface 43 outputs the same image sensor Im to two directions, and this operation is implemented by switching devices or multiplexer. - In the present disclosure, an auto exposure control signal is used to control, for example, an exposure interval of the
image sensor 21, light source intensity and a gain value to change average brightness of the image frame generated by theimage sensor 21 to be within a suitable range. - In other embodiments, the tag indicates a simple analyzed result of the first image frame Im1, e.g., indicating the first image frame Im1 containing a face, human skin color, a human shape object or a vehicle. The processor of the
back end circuit 9 has stronger calculation ability, and said processor performs the operation requiring more calculation such as performing the ID recognition or license plate recognition according to the second image frame Im2. - As mentioned above, in the conventional security monitoring system, a back end circuit performs both the image recording and the feature tagging, and the image sensor outputs image frames having only one size to the back end circuit for the image recording. Accordingly, the present disclosure further provides a photographing device generating image frames of two sizes (e.g. referring to
FIG. 2 ) that recognizes a triggering object in a low resolution image frame at first and then outputs a tagged high resolution image frame to an external back end circuit for the image recording. As the recorded successive images have already contained the feature tag in the data packet, the back end circuit needs not to perform the feature tag anymore. - Although the disclosure has been explained in relation to its preferred embodiment, it is not used to limit the disclosure. It is to be understood that many other possible modifications and variations can be made by those skilled in the art without departing from the spirit and scope of the disclosure as hereinafter claimed.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/988,059 US20230074088A1 (en) | 2019-06-05 | 2022-11-16 | Photographing device capable of outputting tagged image frame |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/431,788 US10757358B1 (en) | 2019-06-05 | 2019-06-05 | Photographing device having two output interfaces |
US16/924,285 US11212484B2 (en) | 2019-06-05 | 2020-07-09 | Photographing device outputting tagged image frames |
US17/515,575 US11533449B2 (en) | 2019-06-05 | 2021-11-01 | Photographing device having two output interfaces |
US17/988,059 US20230074088A1 (en) | 2019-06-05 | 2022-11-16 | Photographing device capable of outputting tagged image frame |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/515,575 Continuation US11533449B2 (en) | 2019-06-05 | 2021-11-01 | Photographing device having two output interfaces |
Publications (1)
Publication Number | Publication Date |
---|---|
US20230074088A1 true US20230074088A1 (en) | 2023-03-09 |
Family
ID=72140909
Family Applications (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/431,788 Active US10757358B1 (en) | 2017-12-26 | 2019-06-05 | Photographing device having two output interfaces |
US16/924,285 Active US11212484B2 (en) | 2017-12-26 | 2020-07-09 | Photographing device outputting tagged image frames |
US17/515,575 Active US11533449B2 (en) | 2019-06-05 | 2021-11-01 | Photographing device having two output interfaces |
US17/988,059 Pending US20230074088A1 (en) | 2019-06-05 | 2022-11-16 | Photographing device capable of outputting tagged image frame |
Family Applications Before (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/431,788 Active US10757358B1 (en) | 2017-12-26 | 2019-06-05 | Photographing device having two output interfaces |
US16/924,285 Active US11212484B2 (en) | 2017-12-26 | 2020-07-09 | Photographing device outputting tagged image frames |
US17/515,575 Active US11533449B2 (en) | 2019-06-05 | 2021-11-01 | Photographing device having two output interfaces |
Country Status (2)
Country | Link |
---|---|
US (4) | US10757358B1 (en) |
CN (2) | CN112055152B (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10757358B1 (en) * | 2019-06-05 | 2020-08-25 | Primesensor Technology Inc. | Photographing device having two output interfaces |
US20220165304A1 (en) * | 2020-11-24 | 2022-05-26 | Nvidia Corporation | Trigger-responsive clip extraction based on remote analysis |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150199098A1 (en) * | 2014-01-15 | 2015-07-16 | Samsung Electronics Co., Ltd. | Method for setting image capture conditions and electronic device performing the same |
US20190370980A1 (en) * | 2018-05-30 | 2019-12-05 | Chiral Software, Inc. | System and method for real-time detection of objects in motion |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9633426B2 (en) * | 2014-05-30 | 2017-04-25 | General Electric Company | Remote visual inspection image capture system and method |
JP4870546B2 (en) * | 2006-12-27 | 2012-02-08 | 株式会社岩根研究所 | CV tag video display device with layer generation / selection function |
JP2009272820A (en) * | 2008-05-02 | 2009-11-19 | Konica Minolta Opto Inc | Solid-state imaging device |
US8265817B2 (en) * | 2008-07-10 | 2012-09-11 | Lockheed Martin Corporation | Inertial measurement with an imaging sensor and a digitized map |
US9135483B2 (en) * | 2011-09-09 | 2015-09-15 | Metrologic Instruments, Inc. | Terminal having image data format conversion |
US8866921B2 (en) * | 2012-06-26 | 2014-10-21 | Htc Corporation | Devices and methods involving enhanced resolution image capture |
US9383833B2 (en) * | 2013-04-09 | 2016-07-05 | Pixart Imaging Inc. | Navigation device and power saving method thereof |
JPWO2015104780A1 (en) * | 2014-01-07 | 2017-03-23 | パナソニックIpマネジメント株式会社 | Video imaging device |
US9531959B2 (en) * | 2014-08-29 | 2016-12-27 | Semiconductor Components Industries, Llc | Imaging systems and methods for capturing image data at high scan rates |
CN105611413B (en) * | 2015-12-24 | 2018-10-02 | 小米科技有限责任公司 | A kind of method and apparatus of addition video-frequency band category label |
JP6683012B2 (en) * | 2016-05-24 | 2020-04-15 | 沖電気工業株式会社 | Image processing system, image processing method, and imaging device |
CN106131529B (en) * | 2016-06-30 | 2018-02-27 | 联想(北京)有限公司 | A kind of method of video image processing and device |
US10757358B1 (en) * | 2019-06-05 | 2020-08-25 | Primesensor Technology Inc. | Photographing device having two output interfaces |
CN108718393B (en) * | 2018-08-03 | 2024-04-05 | 思特威(上海)电子科技股份有限公司 | CMOS image sensor and monitoring method thereof |
-
2019
- 2019-06-05 US US16/431,788 patent/US10757358B1/en active Active
- 2019-12-16 CN CN201911293936.7A patent/CN112055152B/en active Active
- 2019-12-16 CN CN202210021342.6A patent/CN114374797A/en active Pending
-
2020
- 2020-07-09 US US16/924,285 patent/US11212484B2/en active Active
-
2021
- 2021-11-01 US US17/515,575 patent/US11533449B2/en active Active
-
2022
- 2022-11-16 US US17/988,059 patent/US20230074088A1/en active Pending
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150199098A1 (en) * | 2014-01-15 | 2015-07-16 | Samsung Electronics Co., Ltd. | Method for setting image capture conditions and electronic device performing the same |
US20190370980A1 (en) * | 2018-05-30 | 2019-12-05 | Chiral Software, Inc. | System and method for real-time detection of objects in motion |
Also Published As
Publication number | Publication date |
---|---|
US11533449B2 (en) | 2022-12-20 |
US10757358B1 (en) | 2020-08-25 |
US20200389619A1 (en) | 2020-12-10 |
CN112055152A (en) | 2020-12-08 |
US20220053158A1 (en) | 2022-02-17 |
US11212484B2 (en) | 2021-12-28 |
CN112055152B (en) | 2022-02-01 |
CN114374797A (en) | 2022-04-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20230074088A1 (en) | Photographing device capable of outputting tagged image frame | |
US11076114B2 (en) | Dynamic vision sensor architecture | |
CN109640007B (en) | Artificial intelligence image sensing equipment | |
US7312821B2 (en) | Time-sliced still image generation | |
US11405581B2 (en) | Motion detection methods and image sensor devices capable of generating ranking list of regions of interest and pre-recording monitoring images | |
US7742072B2 (en) | Monitoring image recording apparatus | |
US20220132028A1 (en) | Imaging apparatus, information processing method, and recording medium | |
US11336870B2 (en) | Smart motion detection device and related determining method | |
US7148919B2 (en) | Capturing a still image and video image during a single exposure period | |
US20220181367A1 (en) | Image sensor with image receiver and automatic image switching | |
KR100953157B1 (en) | Security camera processer | |
JP2565169B2 (en) | Video signal processing method | |
JP2006135838A (en) | Motion detecting device | |
Stevanovic et al. | High-speed camera with embedded FPGA processing | |
US10750077B1 (en) | Camera system with multiple camera | |
US20230162526A1 (en) | Fingerprint recognition device, readout circuit and operating method of fingerprint recognition device | |
KR20110115315A (en) | Photographing apparatus having image sensor | |
JPH1127654A (en) | Monitoring camera and monitoring system | |
WO2021152437A1 (en) | System and method for capturing and analysing images and/or videos in low light condition | |
JPH057364A (en) | Picture monitoring device | |
KR20090115649A (en) | Security camera processer | |
WO2024008305A1 (en) | An image sensor system, a camera module, an electronic device and a method for operating a camera module for detecting events using infrared | |
JP2004289766A (en) | High-speed imaging apparatus | |
JPH0514890A (en) | Image monitoring device | |
KR20000041522A (en) | Image data compressing method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: PIXART IMAGING INC., TAIWAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PRIMESENSOR TECHNOLOGY INC.;REEL/FRAME:061790/0407 Effective date: 20201016 Owner name: PIXART IMAGING INC., TAIWAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WU, CHIH-HUAN;YEN, WEN-CHENG;REEL/FRAME:061790/0320 Effective date: 20190422 Owner name: PRIMESENSOR TECHNOLOGY INC., TAIWAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WU, CHIH-HUAN;YEN, WEN-CHENG;REEL/FRAME:061790/0320 Effective date: 20190422 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |