JP2006033160A - Imaging system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging system capable of transferring an image with excellent image quality in matching with the state of a camera and the state of a situation required by a user even in a transmission system with a small transmission capacity. <P>SOLUTION: In the imaging system wherein an imaging apparatus provided with a communication means is connected to an information processing apparatus with a communication means by using the communication means, the imaging apparatus is configured to include: a means that uses an imaging element 13 to convert an object image projected by a photographing lens 11 into an electric signal and provides an output of an image as a video signal; an image trimming means; an image compression means; a means for storing compressed image data; and a means for controlling the number of frames for transferring the image per unit time. The information processing apparatus is configured to include: a means for expanding the compressed image data; a means for overlapping a plurality of images; and a means for displaying the image. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to image transmission control of an imaging system.

  In recent years, digital cameras and video cameras such as Web cameras have been connected to image processing devices that have communication functions and information processing devices that have communication functions by means of communication functions, realizing live image display, image file transfer, and remote capture functions. Yes.

  In recent years, with the improvement in performance of solid-state imaging devices (CCDs and the like), the size of the output image and the number of frames per unit time of the entire screen are increasing. Along with these factors, the amount of communication data is also increasing.

JP-A-10-155100

  In the case of a transmission system with a small transmission bandwidth, the image is encoded and the amount of data is reduced as much as possible to transfer a low image. However, in a transmission system that still cannot transfer all images, a frame drop-down image is transferred, but the motion is smooth. There has been a problem that the resolution has deteriorated and the resolution has also deteriorated. In order to solve this problem, Japanese Patent Laid-Open No. 10-66074 has proposed that the amount of motion of image information to be transferred is detected, and frame dropouts or thinned-out images are transferred according to the detection results. There is a problem that the image around the subject required by the user becomes low accuracy.

  Japanese Patent Application Laid-Open No. 2000-217108 proposes to control the amount of data for encoding the image of each part in order to transmit the high-quality image to the target part of the image and transmit the other part with low image quality. However, it is not possible to respond to a request to change the image quality depending on the state of the camera and the state of the situation. For example, if the image is out of focus, the user will not feel uncomfortable even with a low image, and only the image around the subject captured at the time of focusing will be focused on the user. It is not possible to respond to a request to notify that the in-focus state is quicker and clearer.

  In addition, when the transmission medium is wireless, the transmission bandwidth changes from time to time depending on the radio wave condition or the like, but there is a problem that an image required by the user cannot be transmitted dynamically.

  Furthermore, in the case of a multi-function information processing device, the processing speed of the transferred data changes from time to time depending on the state of various information processing, etc., but an image required by the user can be requested from the imaging device and displayed.

  The present invention solves such problems and provides an imaging system capable of transferring an image with a good image quality in accordance with the state of a camera and situation required by a user even in a transmission system having a small transmission capacity. is there.

  In an imaging system in which an imaging apparatus having a communication unit and an information processing apparatus having a communication unit are connected using the communication unit, the imaging apparatus converts an object image projected by a photographing lens into an electrical signal by an imaging element. A means for outputting an image as a video signal; a means for trimming the output image; a means for compressing the image; a means for storing the compressed image data; and a means for controlling the number of frames transferred per unit time. It is a configuration. The information processing apparatus includes a unit that decompresses compressed image data, a unit that overlaps a plurality of images, and a unit that displays an image. The image captured by the imaging device is divided into the image of the window of interest and the other images, the compression rate is changed for each image, the compression processing is performed, the data is transferred by changing the frame rate for each image, and the information The image of the window of interest and the other images are received by the processing device, and the images are expanded and overlapped for display.

  According to the present invention, an imaging system in which an imaging apparatus having a communication unit and an information processing apparatus having a communication unit are connected via a communication unit, and a window of interest is generated from a captured image according to a communication state, a shooting state, and the like, By changing the compression rate / frame rate for each window, it is possible to provide a high-accuracy image that the user originally seeks in the situation / situation. For example, when the subject is out of focus, the user will not feel uncomfortable even with a low image. It is possible to notify that the focus has been achieved more quickly and vividly.

  Even in the case of a system having a small transmission bandwidth, it is possible to transfer a high-quality image and express a smooth movement by effectively using a limited transmission capacity. Furthermore, when the transmission medium is wireless, the transmission bandwidth changes as needed depending on the radio wave condition, etc., but the image required by the user can be transmitted dynamically.

  Furthermore, in the case of a multi-function information processing device, the processing speed of the transferred data changes from time to time depending on the state of various information processing, etc., but an image required by the user can be requested from the imaging device and displayed.

  Embodiments of the present invention will be described below with reference to the accompanying drawings.

  FIG. 5 shows a configuration diagram when the imaging apparatus and the information processing apparatus of the present invention are connected.

  Reference numeral 503 denotes an image pickup apparatus, and reference numeral 504 denotes an information processing apparatus. The imaging device 503 projects a subject 501 with a photographing lens, converts the subject 501 into an electric signal with an imaging element, and displays an image on an electronic viewfinder (EVF) 502 as a video signal. The communication means 500 has various communication functions such as RS232C, USB, IEEE 1394, modem, wired communication such as LAN, infrared communication such as IrDA, and wireless communication such as Bluetooth and 802.11b. The remote capture function can be realized by transferring the image to the information processing apparatus and displaying the image on the display of the information processing apparatus, or by transmitting a command from the information processing apparatus to the imaging apparatus.

  FIG. 6 is a diagram showing the configuration of the information processing apparatus shown in FIG.

  Reference numeral 607 denotes an information processing apparatus, such as a PC, a PDA, a remote controller, or a mobile phone.

  Reference numeral 601 denotes an operation means for inputting various operation instructions, and is composed of a plurality of combinations such as a keyboard, a mouse, and a touch panel. Reference numeral 602 denotes an image display unit including a TFT CRT and the like, which displays image data and the like. If the image data picked up by the image pickup apparatus is displayed at any time, the electronic viewfinder (EVF) function of the information processing apparatus can be realized.

  Reference numeral 603 denotes a compression / decompression unit that compresses and decompresses image data by adaptive discrete cosine transform (ADCT), wavelet transform, and the like, and 605 overlaps a plurality of image data. A frame rate 606 can be controlled at a communication partner.

  Reference numeral 604 denotes a communication means, which has various communication functions such as RS232C, USB, IEEE1394, modem, wired communication such as LAN, infrared communication such as IrDA, wireless communication such as Bluetooth, 802.11b, or the like. A configuration including a plurality of the above may be used.

  FIG. 4 is a block diagram illustrating a configuration example of the imaging apparatus illustrated in FIG.

  Reference numeral 11 denotes a photographing lens, 12 denotes a shutter having a diaphragm function, 13 denotes an image sensor that converts an optical image into an electric signal, and 15 denotes an A / D converter that converts an analog signal output from the image sensor 13 into a digital signal. A timing generation circuit 14 supplies a clock signal and a control signal to the image sensor 13 and the A / D converter 15 and is controlled by the memory control circuit 17 and the system control 31.

  An image processing circuit 16 performs predetermined pixel interpolation processing and color conversion processing on the data from the A / D converter 15 or the data from the memory control circuit 17. The image processing circuit 16 performs predetermined calculation processing using the captured image data, and the system control circuit 35 controls the exposure control means 31 and the distance measurement control means 32 based on the obtained calculation result. TTL (through-the-lens) AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash pre-issuance) processing are performed. Further, predetermined arithmetic processing is performed using the captured image data, and TTL AWB (auto white balance) processing is also performed based on the obtained arithmetic result.

  A memory control circuit 17 controls the A / D converter 15, the timing generation circuit 14, the image processing circuit 16, the image display circuit 19, the memory 20, and the compression / decompression circuit 21.

  The image display circuits 19 and 18 are image display units composed of TFT LCDs or the like, and display image data written in the memory 20 is displayed on the image display unit 18 via the memory control circuit 17. An electronic viewfinder (EVF) function can be realized by displaying image data captured using the image display unit 18 at any time.

  Reference numeral 20 denotes a memory for storing captured still images and moving images, and has a storage capacity sufficient to store a predetermined number of still images and a moving image for a predetermined time. As a result, even when a plurality of images are continuously captured, a large amount of image writing can be performed on the memory 20 at high speed. The memory 20 can also be used as a work area for the system control circuit 35.

  A compression / decompression circuit 21 compresses / decompresses image data by adaptive discrete cosine transform (ADCT), wavelet transform, etc., and reads the image stored in the memory 20 to perform compression processing or decompression processing. Is written into the memory 20.

  A frame rate control circuit 23 controls a frame rate at which an image stored in the memory 20 is displayed 18 via the memory control circuit 17. Further, it is possible to control the frame rate at which the image stored in the memory 20 is transferred to the communication partner via the communication circuit 22.

  Reference numeral 31 denotes exposure control means for controlling the shutter 12 having a diaphragm function, and has a flash light control function in cooperation with the flash 34. Reference numeral 32 denotes distance measuring control means for controlling the focusing of the photographing lens 11, and 33 denotes zoom control means for controlling zooming of the photographing lens 11. A flash 34 has an AF auxiliary light projecting function and a flash light control function.

  Reference numeral 35 denotes a system control circuit for controlling the entire image pickup apparatus, and reference numeral 30 denotes a memory for storing operation constants, variables, and the like of the system control circuit 35. Reference numeral 29 denotes an electrically erasable / recordable nonvolatile memory that stores an operation program, parameters, and the like of the system control circuit 35.

  Reference numerals 26, 27, and 28 denote operation means for inputting various operation instructions of the system control circuit 35, and are constituted by a plurality of combinations such as switches, dials, and touch panels.

  22 is a communication means, which has various communication functions such as wired communication such as RS232C, USB, IEEE1394, modem, LAN, infrared communication such as IrDA, wireless communication such as Bluetooth, 802.11b, etc. A configuration including a plurality of the above may be used.

  Reference numeral 25 denotes a recording medium such as a memory card or a hard disk.

  An embodiment of claim 1 of the present invention will be described below with reference to the drawings and flowcharts.

  FIG. 1 is a screen configuration diagram of a window when a captured image is displayed on an information processing apparatus connected to an electronic viewfinder (EVF) or an imaging apparatus, for example, a display of a PC. The window is divided into a window of interest 100 and a window 101 other than the window of interest, and each window is compressed at a different compression rate and transferred at different frame rates.

  FIGS. 2-1 and 2-2 are flowcharts showing the operation of the imaging apparatus / information processing apparatus that changes the compression rate / frame rate for each window in accordance with the shooting state and the like. The operation will be described in detail.

  FIG. 2A is a flowchart illustrating the operation on the imaging apparatus side.

  When the imaging apparatus and the information processing apparatus are connected and the user starts communication by a GUI operation or the like, a captured image is acquired as shown in step 201. In step 202, the photographing state is acquired and evaluated. The photographing state includes a focus state, an exposure state, a flash pre-issue state, a white balance state, the number of photographing pixels, an image compression rate, an operation from the information processing apparatus, and the like. In the steady state, i.e., in the state where only the process of displaying the captured image without any operation of the imaging device is performed by the evaluation, the captured image is compressed at the medium compression rate at step 203 and the medium frame rate at step 204. Transfer with.

  If the communication response is regarded as important by evaluation, for example, during preparation for photographing, the captured image is compressed at a high compression rate at step 205 and transferred at a high frame rate at step 206. Further, communication response is regarded as important by evaluation, but a high-accuracy image is required around the subject. For example, at the time of shooting, the window of interest is transferred in step 207 as to whether it is a frame for transferring the entire screen of the captured image. Determine if it is a frame. In the case of a frame for transferring the entire screen (in the case of [Yes]), the entire screen is compressed at a high compression rate in step 208 and the data is transferred in step 209.

  Conversely, in the case of a frame for transferring the window of interest (in the case of [No]), in step 210, the captured image is trimmed to match the window to generate the window of interest. In step 211, coordinate data is stored. The window image generated at step 212 is compressed at a low compression rate, and the window image and coordinate data are transferred at step 213. Also, control is performed to transfer at a high frame rate when transferring all the images and at a low frame rate when transferring the window of interest. After the data transfer, the processing from step 201 is repeated. Data representing each shooting state is also transferred.

  FIG. 2B is a flowchart illustrating the operation on the information processing apparatus side.

  When the imaging device and the information processing device are connected and the user starts communication by a GUI operation or the like, as shown in step 220, the presence / absence of an operation on the imaging device is determined, and if there is (if [Yes]), the operation is performed. Transfer in step 221. In step 222, data from the imaging device is acquired. It is determined whether the data acquired at 223 is full screen data or target window / coordinate data. In the case of full screen data (in the case of [Yes]), the data is decompressed in step 224 and stored in the memory 1. In the case of the window of interest (in the case of [No]), the data is expanded in step 225 and stored in the memory 2. In step 226, the state is determined. Although the response is important, if a high-precision image is required around the subject, the window image stored in the memory 2 is overlapped with all the images based on the coordinate data. In step 228, an image is displayed.

  FIG. 3 is a sequence diagram showing exchange of data transfer by the operation of the imaging apparatus / information processing apparatus described in FIGS. 2-1 and 2-2.

  In step 302, the imaging apparatus and the information processing apparatus are connected, and the user starts communication by a GUI operation or the like. In step 303, since the photographing state is a steady state, the captured image data compressed at the medium compression rate is transferred from the image capturing apparatus 301 to the information processing apparatus 300 at the medium frame rate. In step 305, the information processing apparatus 300 notifies the imaging apparatus 301 that the relays preparation has started by the user starting preparations for the relays through a GUI operation or the like of the information processing apparatus 300. At this time, since it is necessary to place importance on the response, in step 306, the captured image data compressed at a high compression rate is transferred from the imaging device 301 to the information processing device 300 at a high frame rate.

  In step 308, the information processing apparatus 300 is notified from the imaging apparatus 301 that the relays preparation of the imaging apparatus 301 has been completed. At this time, since the response is regarded as important, a high-accuracy image is necessary around the subject. Therefore, in step 309, all captured image data compressed at a high compression rate is transferred. In 310, the window image data of interest compressed at a low compression rate is transferred. All captured images are transferred at a medium frame rate, and the window of interest is transferred at a low frame rate.

  As described above, by changing the compression rate of captured image data according to the shooting state and changing the transfer frame rate, the limited transmission capacity of communication can be used effectively, and the captured image can be used. The image around the subject required by the user can be expressed with high accuracy while smoothly expressing the movement of the subject.

  An embodiment of claim 12 of the present invention will be described below with reference to the drawings and flowcharts.

  FIG. 7 is an example of a flowchart showing a process of dynamically calculating and transferring the compression rate, frame rate, and size of the window of interest according to the transmission capacity when the communication state changes.

  When the communication state changes, in step 700, the communication state is checked. If the communication is disabled (in the case of [No]), the communication process is terminated. If communication is possible (in the case of [Yes]), the transmission capacity is calculated in step 701. In step 702, the compression rate / frame rate of all image data and the size of the window of interest and the compression rate / frame rate of the data of the window of interest are set according to the transmission capacity. In step 703, image processing is performed at a compression rate in which all images and the window of interest are set, and in step 704, data is transferred in accordance with the frame rate.

  FIG. 8 is a sequence diagram showing exchange of data transfer by an operation of dynamically changing the compression rate, the frame rate, and the size of the window of interest in accordance with the communication state described in FIG.

  Negotiation is performed to connect the imaging device 801 and the information processing device 800, and if the data communication is possible and the communication state including the radio wave state is good, the transmission capacity increases. All image data is transferred at a high frame rate. In step 803, the size of the window of interest is enlarged, and the window of interest data processed at a low compression rate is transferred at a medium frame rate.

  If the radio wave condition deteriorates and the communication state deteriorates, the transmission capacity decreases. However, if a high-accuracy image is required around the subject, in step 804, all image data processed at a high compression rate is transferred at a medium frame rate. In 805, the size of the window of interest is reduced, and the window data of interest processed at a low compression rate is transferred at a low frame rate.

  As described above, when the transmission medium is wireless, the transmission bandwidth changes as needed depending on the radio wave condition and the like, but the image required by the user can be transmitted dynamically.

  An embodiment of claim 16 of the present invention will be described below with reference to flowcharts.

  FIG. 9 is an example of a flowchart showing a process for requesting the imaging apparatus to change the compression rate, frame rate, window size of interest, etc. of the image data according to the calculated transmission capacity when the state of the information processing apparatus changes. is there.

  In step 900, image data transferred from the imaging apparatus is acquired, and in step 901, the data is stored in a buffer in the information processing apparatus. In step 902, the state of the buffer storing the data is checked, and if it is determined that the buffer is not full (in the case of [No]), the image data compressed and transferred in step 904 is decompressed, and in step 905, the full screen image is displayed. An image in which the window image of interest is overlapped is generated, and in step 906, the image is displayed on the image display unit of the information processing apparatus. If it is determined in step 902 that the buffer is full (in the case of [Yes]), the size, compression rate, frame rate, etc. of the window of interest are calculated in step 903, and the window of interest calculated for the imaging apparatus is calculated in step 904. Request to change the size, compression rate, frame rate, etc.

  Here, the state of the buffer for storing data is listed. However, if the information processing apparatus is a PC, the state of the memory of the PC, the state of a large-capacity recording medium such as an HDD, the state of communication with other than the imaging device, the CPU The size, compression rate, frame rate, and the like of the window of interest are calculated in accordance with the processing state, and a request for changing the size, compression ratio, frame rate, etc. of the calculated window of interest is made to the imaging apparatus.

  In response to the request from the information processing apparatus, the imaging apparatus performs trimming processing on the captured image in accordance with the size of the requested target window, performs compression processing at the requested compression rate, and performs data transfer at the requested frame rate.

  As described above, in the case of a multi-function information processing device, the processing speed of the transferred data changes from time to time depending on the status of various information processing, etc., but requests and displays an image required by the user from the imaging device. Can do.

It is a window screen block diagram of an embodiment of the invention. 3 is a flowchart illustrating an operation procedure of the imaging apparatus according to the first embodiment. 3 is a flowchart illustrating an operation procedure of the information processing apparatus according to the first embodiment. FIG. 3 is a sequence diagram illustrating exchange of data transfer by the operation of the imaging apparatus / information processing apparatus according to the first embodiment. It is a block diagram of an imaging device showing the configuration of the present invention. 1 is a connection diagram between an imaging apparatus and an information processing apparatus showing a configuration of the present invention. 1 is a configuration diagram of an information processing apparatus according to the present invention. It is a flowchart which shows the operation | movement procedure of the imaging device and information processing apparatus of an Example. It is the sequence diagram which showed the exchange of the data transfer by operation | movement of the imaging device and information processing apparatus of an Example. It is a flowchart which shows the operation | movement procedure of the information processing apparatus of an Example.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Shooting lens 12 Shutter 13 Image sensor 14 Timing generation circuit 15 A / D converter 16 Image processing circuit 17 Memory control circuit 18 Image display part 19 Image display circuit 20 Memory 21 Image compression / expansion circuit 22 Communication means 23 Frame rate control circuit 24 interface 29 ROM
30 RAM
31 Exposure control means 32 Distance control means 33 Zoom control means 34 Flash

Claims (18)

In an imaging system in which an imaging apparatus including a communication unit and an information processing apparatus including a communication unit are connected by the communication unit,
The image pickup apparatus includes: a means for converting a subject image projected by a photographing lens into an electric signal by an image pickup device and outputting an image as a video signal; a means for trimming the output image; a means for compressing the image; Means for storing the image data, and means for controlling the number of frames for transferring the image per unit time. The information processing apparatus includes a means for expanding the compressed image data and a plurality of images. A means for displaying the image and a means for displaying the image, dividing the image captured by the imaging device into an image of one or more windows of interest and other images, and changing the compression rate for each image and performing compression processing The frame rate for each image is changed and data is transferred, and the information processing device receives the image of the window of interest and other images, decompresses each image, and overlaps Imaging system, characterized in that to cause the display.   The imaging system according to claim 1, wherein compression processing is performed by changing a compression rate of a window of interest and other images according to a shooting state, and data transfer is performed by changing a frame rate.   The imaging system according to claim 1 or 2, wherein in an out-of-focus state, all images are compressed at a high compression rate and data is transferred at a low frame rate.   In the focused state, the image of the window of interest is compressed at a low compression rate and transferred at a low frame rate, and the other images are compressed at a high compression rate and transferred at a medium frame rate. The imaging system according to claim 1 or 2.   In the AF (autofocus) state, the compression rate and frame rate are automatically set according to the focus evaluation value, compression processing and data transfer are performed, and other images are compressed at a high compression rate. 3. The imaging system according to claim 1, wherein the imaging system performs processing and transfers data at a low frame rate.   In the AE (automatic exposure) state, the image of the window of interest is compressed at a low compression rate and transferred at a low frame rate, and the other images are compressed at a high compression rate and transferred at a low frame rate. The imaging system according to claim 1 or 2.   3. The imaging system according to claim 1, wherein in a state where the photographing apparatus is panned, image processing is performed at a high compression rate and data is transferred at a low frame rate.   When the shooting preparation is complete, the image of the window of interest is compressed at a low compression rate and transferred at a low frame rate, and the other images are compressed at a medium compression rate and transferred at a medium frame rate. The imaging system according to claim 1 or 2, characterized in that   2. A user interface capable of setting weighting to a shooting state, wherein the weighting is used to change the compression rate of all images and the image of the window of interest to perform compression processing, and to transfer data by changing the frame rate. Or the imaging system of 2.   The imaging system according to claim 1, further comprising a user interface capable of setting a compression rate and a frame rate of all images according to a shooting state.   3. The imaging system according to claim 1, further comprising a user interface capable of setting a size, a compression rate, and a frame rate of a window of interest according to a shooting state.   The size of the window of interest is changed in accordance with the communication state, the compression ratio of the window of interest and other images is changed to perform compression processing, and the data transfer is performed by changing the frame rate. Imaging system.   When the communication state is good and the transmission capacity is large, the image of the window of interest is compressed at a low compression rate and transferred at a medium frame rate, and the other images are compressed at a medium compression rate and data at a high frame rate. 13. The system according to claim 1, wherein the system is transferred.   When the communication state deteriorates and the transmission capacity is small, the image size of the window of interest is reduced, compressed at a low compression rate and transferred at a low frame rate, and other images are compressed at a high compression rate, and the medium frame 13. The imaging system according to claim 1, wherein data transfer is performed at a rate.   13. The imaging system according to claim 1, further comprising a user interface capable of setting a compression rate and a frame rate of all images according to a communication state.   13. The imaging system according to claim 1, further comprising a user interface capable of setting a size, a compression rate, and a frame rate of a window of interest according to a communication state.   The size of the window of interest is changed in accordance with the state of the information processing apparatus, the compression ratio of the window of interest and the other image is changed to perform compression processing, and the data transfer is performed by changing the frame rate. The imaging system according to 1.   2. The imaging system according to claim 1, wherein when photographing, the image is compressed at a low compression rate, a highly accurate image is recorded, and data is transferred.

Images