JP2017050818A - Image processing method and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the quality of an image to be displayed on a display device by adjusting order in which a plurality of images taken by a plurality of imaging devices are processed.SOLUTION: In an image processing method in which a plurality of photographic image taken by a plurality of imaging devices are displayed on a display device as display images, a capture time indicating the time at which the photographic images are stored in a storage part is recorded so as to correspond to each of the imaging devices; drawing processes for displaying on the display device are performed in order for the respective photographic images stored in the storage device; a drawing start time indicating the time at which the drawing process is started is recorded so as to correspond to each of the imaging devices; a drawing completion time indicating the time at which a drawing process is completed is recorded so as to correspond to each of the imaging devices; and execution order of the drawing processes for the respective photographic images when the subsequent display images are displayed on the display device is determined on the basis of the capture time, drawing start time and drawing completion time, all of which have been already recorded.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing method and an image processing program.

  An image processing apparatus that displays a plurality of images taken by a plurality of imaging devices on one screen has been proposed (see, for example, Patent Document 1). For example, the image processing device stores a plurality of images captured by a plurality of imaging devices in a storage unit, executes a process of combining the plurality of images stored in the storage unit into one image, and displays the combined image on the display device. .

  In the case where there is one imaging device, a method for adjusting the start timing of the imaging operation so that the time from when the imaging operation by the imaging device starts until the image is displayed has been proposed (for example, Patent Document 2).

  Also, a display device has been proposed that updates the display without causing a decrease in the frame rate when scrolling the displayed image (for example, see Patent Document 3).

JP 2002-302001 A JP 2008-166859 A JP 2013-214034 A

  When a plurality of images captured by a plurality of imaging devices are displayed on a display device, the timing at which the plurality of imaging devices capture a subject (hereinafter also referred to as imaging timing) is not necessarily the same. Therefore, when the image processing apparatus sequentially processes each of the plurality of images captured by the plurality of imaging apparatuses in a predetermined processing unit (for example, one frame of a moving image captured by the imaging apparatus), the order of processing the images Depending on the case, the quality of the image displayed on one screen of the display device may differ.

  The quality of the image is, for example, a delay time until the image is displayed (average of delay times of a plurality of images displayed on one screen of the display device), or a plurality of images displayed on one screen of the display device. It varies depending on the magnitude of the shooting timing deviation.

  In one aspect, the image processing method and the image processing program disclosed herein improve the quality of an image displayed on a display device by adjusting the order of processing a plurality of images captured by a plurality of imaging devices. With the goal.

  According to one aspect, with reference to a storage unit that stores a plurality of captured images respectively captured by a plurality of imaging devices for each frame of each imaging device, the plurality of captured images are displayed as display images on a display device for a predetermined time. In the image processing method in which images are sequentially displayed at intervals, capture times indicating times when a plurality of captured images are stored in the storage unit are recorded in association with each of the plurality of imaging devices, and a plurality of shootings stored in the storage unit are recorded. For each of the images, a drawing process for displaying on the display device is sequentially executed, and a drawing start time indicating a time when the drawing process is started is recorded in association with each of the plurality of imaging devices, and the drawing process is performed. The drawing completion time indicating the time when the image is completed is recorded in association with each of the plurality of imaging devices, and the execution order of the drawing processing for each of the plurality of captured images when the next display image is displayed on the display device is Already recorded Yapucha time, be determined on the basis of the drawing start time and drawing completion time.

  According to another aspect, with reference to a storage unit that stores a plurality of captured images respectively captured by a plurality of imaging devices for each frame of each imaging device, a plurality of captured images are displayed as predetermined images on a display device. In an image processing program that causes a computer to perform image processing that is sequentially displayed at time intervals, capture times indicating times when a plurality of captured images are stored in a storage unit are recorded in association with each of the plurality of imaging devices, and stored. For each of the plurality of captured images stored in the unit, drawing processing for displaying on the display device is executed in order, and a drawing start time indicating the time when the drawing processing is started is assigned to each of the plurality of imaging devices. The drawing completion time indicating the time when the drawing process is completed is recorded in association with each other and recorded in association with each of the plurality of imaging devices, and each of the plurality of captured images when the next display image is displayed on the display device The execution order of the drawing action for already recorded captured time is determined based on the drawing starting time and the drawing completion time.

  The image processing method and the image processing program of the present disclosure can improve the quality of an image displayed on a display device by adjusting the order of processing a plurality of images captured by a plurality of imaging devices.

It is a figure which shows one Embodiment of an image processing method and an image processing program. It is a figure which shows an example of operation | movement of the image processing apparatus shown in FIG. It is a figure which shows another embodiment of the image processing method and an image processing program. It is a figure which shows an example of the information referred by the processor shown in FIG. It is a figure which shows an example of the operation timing of the image processing system shown in FIG. FIG. 4 is a diagram illustrating an example of an operation of the processor illustrated in FIG. 3. It is a figure which shows an example of the order determination process shown in FIG. It is a figure which shows another example of the order determination process shown in FIG. It is a figure which shows an example of the simulation result at the time of making the square average of display delay time into an evaluation value. It is a figure which shows an example of the simulation result at the time of setting dispersion | variation of capture time as an evaluation value. It is a figure which shows an example of the hardware constitutions of the image processing system shown in FIG.

  Hereinafter, embodiments will be described with reference to the drawings.

  FIG. 1 shows an embodiment of an image processing method and an image processing program. An image processing system 10 illustrated in FIG. 1 includes a plurality of imaging devices 50 (50a, 50b, 50c), an image processing device 20, and a display device 60. The imaging device 50 (50a, 50b, 50c) is, for example, a digital camera, and outputs captured images CIMG (CIMGa, CIMGb, CIMGc) captured at a predetermined frame rate to the image processing device 20. The image processing device 20 is, for example, a processor such as a CPU (Central Processing Unit).

  The image processing apparatus 20 includes a storage unit 30, a drawing processing unit 40, a capture time recording unit 42, a drawing start time recording unit 44, a drawing completion time recording unit 46, and an order determination unit 48. Note that the image processing apparatus 20 may be realized only by hardware, or may be realized by controlling hardware by software such as an image processing program. For example, the drawing processing unit 40, the capture time recording unit 42, the drawing start time recording unit 44, the drawing completion time recording unit 46, and the order determination unit 48 may be realized by a processor executing an image processing program.

  The image processing device 20 displays a plurality of captured images CIMG captured by the plurality of imaging devices 50 as display images DIMG at a predetermined time interval on one screen of the display device 60. For example, each captured image CIMG is an image for one frame of each imaging device 50, and the display image DIMG is an image in which the captured images CIMGa, CIMGb, and CIMGc are arranged at predetermined locations in the screen of the display device 60. is there.

  The storage unit 30 stores the captured images CIMGa, CIMGb, and CIMGc received from the imaging devices 50a, 50b, and 50c for each frame of each imaging device 50. For example, the storage unit 30 is allocated with areas for storing the captured images CIMG for three frames for the captured images CIMGa, CIMGb, and CIMGc, respectively. By holding the captured image CIMG for three frames, it is possible to reduce the waiting time for a reference request for the captured image CIMG.

  Note that the size of the area allocated for each of the captured images CIMGa, CIMGb, and CIMGc is not limited to a size that can store the captured images CIMG for three frames. For example, the storage unit 30 may be assigned areas for storing the captured images CIMG for two frames for the captured images CIMGa, CIMGb, and CIMGc, respectively.

  The capture time recording unit 42 records the capture times indicating the times at which the plurality of captured images CIMG respectively captured by the plurality of imaging devices 50 are stored in the storage unit 30 in association with each of the plurality of imaging devices 50. . For example, when the image processing apparatus 20 is realized by a processor, the capture time recording unit 42 registers the capture time based on an interrupt or the like that occurs when the captured image CIMG for one frame is stored in the storage unit 30. Record in etc. Thereby, the capture time is recorded in a register or the like for each captured image CIMG (that is, for each imaging device 50).

  The drawing processing unit 40 sequentially executes drawing processing for displaying on the display device 60 for each of the plurality of captured images CIMG stored in the storage unit 30. For example, the drawing processing unit 40 executes drawing processing for each of the plurality of captured images CIMG according to the processing order determined by the order determining unit 48. In the period before the processing order (drawing process execution order) is determined by the order determining unit 48, the drawing processing unit 40 performs drawing processing on each of the plurality of captured images CIMG according to a predetermined default processing order. Execute.

  For example, the drawing processing unit 40 notifies the drawing start time recording unit 44 of the drawing start time indicating the time at which the drawing processing is started, and the drawing completion time recording unit 46 indicates the drawing completion time indicating the time at which the drawing processing is completed. Notify

  When the drawing process for one screen of the display device 60 is completed, the drawing processing unit 40 uses a plurality of photographed images CIMGa, CIMGb, and CIMGc for which the drawing process has been completed as one display image DIMG for a predetermined synchronization signal or the like. Output to the display device 60 in synchronization with the timing.

  The drawing start time recording unit 44 records the drawing start time in association with each of the plurality of imaging devices 50. For example, the drawing start time recording unit 44 records the drawing start time notified from the drawing processing unit 40 in a register or the like for each captured image CIMG (that is, for each imaging device 50).

  The drawing completion time recording unit 46 records the drawing completion time in association with each of the plurality of imaging devices 50. For example, the drawing completion time recording unit 46 records the drawing completion time notified from the drawing processing unit 40 in a register or the like for each captured image CIMG (that is, for each imaging device 50).

  For example, the order determination unit 48 determines the execution order of the drawing processing for the plurality of captured images CIMG, and thereby determines which of the moving images captured by each imaging device 50 as the plurality of captured images CIMG used for the display image DIMG. Whether to select a frame is determined for each imaging device 50. For example, the order determination unit 48 sets the execution order of the drawing processing for each of the plurality of captured images CIMG when displaying the next display image DIMG on the display device 60 to be a processing order suitable for a predetermined condition. It is determined using the capture time, drawing start time, and drawing completion time.

  The predetermined condition is, for example, a condition for improving the quality of the display image DIMG. For example, when the image processing device 20 is installed in a system that displays an image of a subject on the display device 60 in real time, reducing the display delay time as much as possible is one of the conditions for improving the quality of the display image DIMG. . The display delay time is, for example, the time from when each captured image CIMG is stored in the storage unit 30 until it is displayed on the display device 60.

  As the display delay time increases, the difference between the state of the subject in the display image DIMG displayed on the display device 60 and the current state of the subject increases. For this reason, for example, when the image processing apparatus 20 is mounted on the back monitor of a car, the longer the display delay time, the longer the alerting to the driver of the car. In other words, by delaying the display delay time as much as possible, it is possible to prevent delays in alerting the driver of the automobile.

  For example, when the predetermined condition is to make the display delay time as small as possible, the order determination unit 48 can average the display delay times (or the mean square) of the plurality of captured images CIMG in the next display image DIMG as much as possible. The execution order of the drawing processing is determined so as to be smaller. Note that the display delay times of the plurality of captured images CIMG in the next display image DIMG are predicted using the already recorded capture time, drawing start time, drawing completion time, and the like.

  Further, the predetermined condition (condition for improving the quality of the display image DIMG) is not limited to making the display delay time as small as possible. For example, the condition for improving the quality of the display image DIMG may be to minimize the shift in capture time of the plurality of captured images CIMG in the display image DIMG. The shift in capture time of the plurality of captured images CIMG in the display image DIMG corresponds to the shift in shooting timing of the plurality of captured images CIMG in the display image DIMG. The photographing timing is a timing at which the imaging device 50 photographs a subject.

  For example, when the imaging device 50 is mounted on a car or the like or moves, or when the subject moves, the plurality of captured images CIMG are displayed as one display image DIMG as the difference in shooting timing of the plurality of imaging devices 50 increases. When this happens, image distortion will increase.

  Therefore, when a plurality of photographed images CIMG photographed by the plurality of imaging devices 50 are displayed as one display image DIMG within one screen, the larger the deviation in the photographing timing of the plurality of imaging devices 50, the larger the display image DIMG. The quality may be reduced. Note that the shift in shooting timing of the plurality of captured images CIMG in the display image DIMG is adjusted by adjusting the shift in capture time of the plurality of captured images CIMG in the display image DIMG. That is, the quality of the display image DIMG can be improved by minimizing the shift in capture time of the plurality of captured images CIMG in the display image DIMG.

  When the predetermined condition is to minimize the shift in capture time of the plurality of captured images CIMG, the order determination unit 48 minimizes the shift in capture time of the plurality of captured images CIMG in the next display image DIMG. As described above, the execution order of the drawing processing is determined. Note that a shift in the capture time of the plurality of captured images CIMG in the next display image DIMG is predicted using the already recorded capture time, drawing start time, drawing completion time, and the like.

  As described above, the order determination unit 48 performs the drawing process based on the already recorded capture time, drawing start time, and drawing completion time so that the processing order is suitable for the condition for improving the quality of the display image DIMG. Determine the execution order. The processing order determined by the order determination unit 48 (drawing processing execution order) is notified to the drawing processing unit 40.

  The display device 60 is a monitor such as a liquid crystal display, for example, and displays the display image DIMG received from the drawing processing unit 40 at a predetermined cycle (for example, the cycle of the vertical synchronization signal). Note that the configurations of the image processing system 10 and the image processing apparatus 20 are not limited to the example shown in FIG. For example, the number of imaging devices 50 may be two, or four or more. For example, the storage unit 30 may be disposed outside the image processing apparatus 20.

  FIG. 2 shows an example of the operation of the image processing apparatus 20 shown in FIG. The operation illustrated in FIG. 2 may be realized only by hardware, or may be realized by controlling the hardware by software. For example, software such as an image processing program may cause the processor to execute the operation shown in FIG. That is, the processor may read the image processing program and execute the operation shown in FIG. A program that causes a processor (that is, a computer) to execute the operation illustrated in FIG. 2 is an aspect of an image processing program.

  In step S100, the image processing apparatus 20 first processes a captured image CIMG whose processing order determined by the order determination unit 48 is the first among the captured images CIMGa, CIMGb, and CIMGc. Is selected as a captured image CIMG. In the period before the order determination unit 48 determines the processing order (for example, before the first display image DIMG is displayed on the display device 60), the image processing device 20 has a predetermined default processing order. Based on this, the captured image CIMG in the processing order j (j = 1) is selected.

  Next, in step S110, the capture time recording unit 42 captures the capture time indicating the time when the captured image CIMG of the processing order j is stored in the storage unit 30 to the imaging device 50 that has captured the captured image CIMG of the processing order j. Correspondingly record in a register or the like.

  Next, in step S120, the drawing processing unit 40 starts drawing processing on the captured image CIMG in the processing order j. After the drawing process is executed, that is, after the process of step S120 is executed, the operation of the image processing apparatus 20 proceeds to step S130.

  In step S130, the drawing start time recording unit 44 registers the drawing start time indicating the start time of the drawing process for the captured image CIMG in the processing order j in association with the imaging device 50 that has captured the captured image CIMG in the processing order j. Record in etc.

  Next, in step S140, the drawing completion time recording unit 46 waits until drawing processing for the captured image CIMG in the processing order j is completed. The drawing completion time recording unit 46 executes the process of step S150 when the drawing process for the captured image CIMG in the processing order j is completed.

  In step S150, the drawing completion time recording unit 46 registers the drawing completion time indicating the drawing processing completion time for the captured image CIMG in the processing order j in association with the imaging device 50 that has captured the captured image CIMG in the processing order j. Record in etc.

  Next, in step S160, the image processing apparatus 20 determines whether the drawing process for one screen has been completed. For example, the image processing apparatus 20 determines whether drawing processing has been completed for all of the captured images CIMGa, CIMGb, and CIMGc. When the drawing process for one screen is completed, the operation of the image processing apparatus 20 proceeds to step S180. On the other hand, when the drawing process for one screen is not completed, the operation of the image processing apparatus 20 proceeds to step S170.

  In step S170, the image processing apparatus 20 selects the captured image CIMG of the processing order j (j = j + 1) to be processed next from the captured images CIMGa, CIMGb, and CIMGc. After the process of step S170 is executed, the operation of the image processing apparatus 20 returns to step S110. That is, the processes in steps S110 to S150 are performed in the processing order determined by the order determining unit 48 until the drawing process for one screen is completed (the default processing order in the period before the order determining unit 48 determines the processing order). ) Is repeated. When the drawing process for one screen is completed, the process of step S180 is executed.

  In step S180, the order determination unit 48 sets the execution order (processing order) of the drawing processing for each of the plurality of captured images CIMG when the next display image DIMG is displayed on the display device 60 to a process suitable for a predetermined condition. Decide to be in order. For example, the order determination unit 48 captures time, drawing start time, drawing completion time, etc. so that the average (or the mean square) of the display delay times of the plurality of captured images CIMG in the next display image DIMG is as small as possible. Is used to determine the execution order of the drawing processing. Alternatively, the order determination unit 48 uses the capture time, the drawing start time, the drawing completion time, and the like so that the shift in the capture time of the plurality of captured images CIMG in the next display image DIMG is as small as possible. The execution order may be determined.

  Next, in step S190, the image processing device 20 detects, for example, the display image DIMG until a vertical synchronization signal or the like corresponding to the timing for displaying the display image DIMG on the display device 60 (hereinafter also referred to as display timing) is detected. Waiting for transfer to the display device 60. That is, the image processing device 20 executes the process of step S200 in synchronization with the display timing of the vertical synchronization signal or the like.

  In step S200, the image processing apparatus 20 outputs a plurality of captured images CIMGa, CIMGb, and CIMGc for which the drawing process has been completed to the display apparatus 60 as one display image DIMG. As a result, the display image DIMG is displayed on the display device 60. After the process of step S200 is executed, the operation of the image processing apparatus 20 returns to step S100, and the process of displaying the next display image DIMG on the display apparatus 60 is executed. Thereby, the display image DIMG is sequentially displayed on the display device 60 in synchronization with the display timing of the vertical synchronization signal or the like.

  Here, for example, depending on the image processing system 10, the shooting timings of the plurality of imaging devices 50 are not necessarily the same. In addition, the shift in shooting timing of the plurality of imaging devices 50 may change during shooting. Therefore, when the execution order of the drawing processing is fixed in an arbitrary order without considering the condition for improving the quality of the display image DIMG, the captured image CIMG is processed in a different order from the fixed order. May improve quality.

  For this reason, the order determination unit 48 captures the capture time so that the order of processing the plurality of captured images CIMG captured by the plurality of imaging devices 50 becomes a processing order suitable for the condition for improving the quality of the display image DIMG. The adjustment is made based on the drawing start time, the drawing completion time, and the like. Thereby, in the operation shown in FIG. 2, the quality of the display image DIMG displayed on the display device 60 is improved as compared with the method in which the execution order of the drawing processing for the plurality of captured images CIMG is fixed. The operation of the image processing apparatus 20 is not limited to the example shown in FIG.

  As described above, in the embodiment shown in FIGS. 1 and 2, the order determination unit 48 captures, draws, and finishes so that, for example, the processing order is suitable for conditions that improve the quality of the display image DIMG. Is used to determine the execution order of the drawing processing. Thereby, the quality of the display image DIMG can be improved.

  FIG. 3 shows another embodiment of the image processing method and the image processing program. Elements that are the same as or similar to the elements described in FIGS. 1 and 2 are given the same or similar reference numerals, and detailed descriptions thereof are omitted. The image processing system SYS illustrated in FIG. 3 includes a plurality of cameras CAM (CAM1, CAM2, CAM3, CAM4), a storage unit MEM (MEM1, MEM2, MEM3, MEM4), a processor PU, and a display device DIS.

  The camera CAM is a digital camera, for example, and is an example of an imaging device. For example, the cameras CAM1, CAM2, CAM3, and CAM4 respectively transfer the captured images CIMG1, CIMG2, CIMG3, and CIMG4 captured at a predetermined frame rate to the storage units MEM1, MEM2, MEM3, and MEM4. Note that the captured images CIMG (CIMG1, CIMG2, CIMG3, and CIMG4) may be transferred from the camera CAM to the storage unit MEM via the processor PU. The number at the end of the code of the camera CAM, the number at the end of the code of the storage unit MEM, and the number at the end of the captured image CIMG (numbers after the code CIMG in FIG. 4 and the like) correspond to each other.

  Each storage unit MEM has three buffers BUF (BUFa, BUFb, and BUFc) that store the captured images CIMG for one frame, and sequentially stores the captured images CIMG in a triple buffer system. The numbers at the end of the codes of the buffers BUFa, BUFb, and BUFc correspond to the numbers at the end of the codes of the storage unit MEM. For example, the buffers BUFa1, BUFb1, and BUFc1 are the buffers BUF included in the storage unit MEM1.

  Hereinafter, the buffers BUFa1, BUFb1, and BUFc1 are also referred to as buffers BUF1, and the buffers BUFa2, BUFb2, and BUFc2 are also referred to as buffers BUF2. Similarly, the buffers BUFa3, BUFb3, and BUFc3 are also referred to as buffers BUF3, and the buffers BUFa4, BUFb4, and BUFc4 are also referred to as buffers BUF4.

  The processor PU is a CPU, for example, and functions as an image processing apparatus by executing an image processing program. The processor PU displays a plurality of captured images CIMG captured by the plurality of cameras CAM as display images DIMG at a predetermined time interval on one screen of the display device DIS. For example, each captured image CIMG is an image for one frame of each camera CAM, and the display image DIMG is an image in which the captured images CIMG1, CIMG2, CIMG3, and CIMG4 are respectively arranged at predetermined locations on the screen of the display device DIS. It is.

  For example, the processor PU includes a capture time recording unit CREC, a reading time recording unit RREC, a drawing completion time recording unit FREC, a display time recording unit VREC, a prediction unit CAL, an order determination unit SEQ, and a drawing processing unit DRAW. The capture time recording unit CREC, the reading time recording unit RREC, the drawing completion time recording unit FREC, the display time recording unit VREC, the prediction unit CAL, the order determination unit SEQ, and the drawing processing unit DRAW are executed by the processor PU executing the image processing program. It is realized by.

  The capture time recording unit CREC records the capture times indicating the times at which the captured images CIMG1-CIMG4 captured by the cameras CAM1-CAM4 are stored in the storage units MEM1-MEM4, in association with each of the plurality of cameras CAM. To do. For example, the capture time recording unit CREC, for each captured image CIMG (that is, the camera), shows a time stamp indicating the capture time based on an interrupt generated when the captured image CIMG for one frame is stored in the storage unit MEM. Recorded in a register or the like every CAM).

  The drawing processing unit DRAW sequentially performs drawing processing for displaying on the display device DIS for each of the captured images CIMG1 to CIMG4 stored in the storage units MEM1 to MEM4. For example, the drawing processing unit DRAW reads a captured image CIMG from the storage unit MEM, and performs a series of processes for executing the drawing process on the captured image CIMG read from the storage unit MEM, in the processing order determined by the order determination unit SEQ. This is executed for each captured image CIMG. In this case, the time when the processor PU reads the captured image CIMG from the storage unit MEM corresponds to the start time of the drawing process.

  When the drawing process for one screen of the display image DIMG is completed, the drawing processing unit DRAW uses the captured images CIMG1, CIMG2, CIMG3, and CIMG4 for which the drawing process has been completed as one display image DIMG in synchronization with a predetermined timing. Output to the display device DIS. For example, the drawing processing unit DRAW outputs the display image DIMG to the display device DIS in synchronization with the vertical synchronization signal.

  The reading time recording unit RREC records the reading time indicating the time when the processor PU reads the captured image CIMG from the storage unit MEM in association with each of the plurality of cameras CAM. The reading time is an example of a drawing start time indicating a time when the drawing process is started. For example, each time the processor PU reads the captured image CIMG from the storage unit MEM, the reading time recording unit RREC records a time stamp indicating the reading time in a register or the like. Thereby, the reading time is recorded in a register or the like for each captured image CIMG (that is, for each camera CAM).

  The drawing completion time recording unit FREC records the drawing completion time indicating the time when the drawing process is completed in association with each of the plurality of cameras CAM. For example, the drawing completion time recording unit FREC records a time stamp indicating the drawing completion time in a register or the like every time drawing processing for the captured image CIMG for one frame is completed. Thereby, the drawing completion time is recorded in a register or the like for each captured image CIMG (that is, for each camera CAM).

  The display time recording unit VREC records a display time indicating the time when the display image DIMG is displayed on the display device DIS. For example, each time the processor PU outputs the display image DIMG to the display device DIS, the display time recording unit VREC records a time stamp indicating the display time in a register or the like.

  The prediction unit CAL uses the already recorded display time, capture time, read time, and drawing completion time to determine the capture time, read time, and drawing completion time corresponding to the next display image DIMG of the plurality of captured images CIMG. Predict for each combination of processing order.

  The order determining unit SEQ captures, reads, and completes the rendering order of the rendering process for each of the plurality of captured images CIMG when the next display image DIMG is displayed on the display device DIS. Determine based on. For example, the order determination unit SEQ calculates an evaluation value corresponding to a predetermined condition for each combination of processing orders based on the capture time, reading time, and drawing completion time predicted by the prediction unit CAL. Then, the order determination unit SEQ determines the execution order of the drawing process when displaying the next display image DIMG on the display device DIS based on the calculated evaluation value.

  Note that the evaluation value corresponding to the predetermined condition is, for example, when the predetermined condition is to make the display delay time as small as possible until the captured image CIMG is stored in the storage unit MEM and displayed on the display device DIS. It is an evaluation value related to the time. For example, when the predetermined condition is to minimize the shift in capture time of the plurality of captured images CIMG as much as possible, the evaluation value corresponding to the predetermined condition is an evaluation value related to variations in capture time.

  As described above, the order determination unit SEQ sets the execution order of the drawing processing for each of the plurality of captured images CIMG when the next display image DIMG is displayed on the display device DIS to be an order suitable for a predetermined condition. And the capture time, the read time, and the drawing completion time.

  The display device DIS is a monitor such as a liquid crystal display, for example, and displays the display image DIMG received in synchronization with the vertical synchronization signal from the drawing processing unit DRAW.

  Note that the configuration of the image processing system SYS is not limited to the example shown in FIG. For example, the storage unit MEM, the processor PU, and the like may be connected via a bus. In this case, for example, the captured image CIMG is transferred from the storage unit MEM to the processor PU via the bus. The number of cameras CAM may be two or three. Alternatively, the number of cameras CAM may be five or more. The storage unit MEM may be provided in the processor PU. In addition, the capture time recording unit CREC, the reading time recording unit RREC, the drawing completion time recording unit FREC, the display time recording unit VREC, the prediction unit CAL, the sequence determination unit SEQ, and the drawing processing unit DRAW may be realized only by hardware. Good.

  FIG. 4 shows an example of information referred to by the processor PU shown in FIG. The camera frame number (m, Gjn) shown in FIG. 4 is the frame number of the captured image CIMG, and the display frame number (n) is the frame number of the display image DIMG.

  The camera operation system information includes a captured image CIMGim, a captured image update period Ui, a processing order Oi, a capture time Cim, and the like. The processor operation system information includes a reading time Rjn, a camera frame number Gjn, a drawing completion time Fjn, a frame drawing completion time Fn, a display time Vn, and the like. First, information on the camera operation system will be described.

  The camera number i indicates an identification number assigned to each camera CAM. For example, the number at the end of the camera CAM code corresponds to the camera number i. The camera CAM1 is a camera CAM with camera number 1. The captured image CIMGim indicates the captured image CIMG of the camera frame number m captured by the camera CAMi. For example, the captured image CIMG1m indicates the captured image CIMG of the mth frame of the moving image captured by the camera CAM1. Note that the captured images CIMG1m, CIMG2m, CIMG3m, and CIMG4m may be described with the symbol m omitted as shown in FIG. 3 when the camera frame number m is not specified.

  The captured image update period Ui indicates a frame period of a moving image captured by the camera CAMi. For example, the captured image update period U1 indicates a frame period of a moving image captured by the camera CAM1.

  The processing order Oi indicates the execution order of the drawing process for each captured image CIMGi when the display image DIMG (hereinafter also referred to as the display image DIMGn) with the display frame number n is displayed on the display device DIS. For example, when the drawing process for the captured image CIMG1 captured by the camera CAM1 is executed fourth in the display frame period Tn (nth display frame period T), the processing order O1 is set to 4. The display frame period T is a period corresponding to the period of the vertical synchronization signal, for example.

  The capture time Cim indicates the time when the captured image CIMGim of the camera frame number m captured by the camera CAMi is stored in the storage unit MEM. For example, the capture time C1m indicates the time when the captured image CIMG1m of the mth frame of the moving image captured by the camera CAM1 is stored in the storage unit MEM1.

  Next, processor operation system information will be described. The processing order j indicates the order in which the processor PU executes the drawing process in each display frame period T. Note that the captured images CIMG1 to CIMG4 for which the drawing process has been executed in the display frame period Tn are displayed on the display device DIS in the display frame period T (n + 1) as the display image DIMGn.

  The camera CAMi corresponding to the processing order j is a camera CAMi whose processing order Oi is j. Therefore, the reading time Rjn, the camera frame number Gjn, and the drawing completion time Fjn are associated with one of the plurality of cameras CAMi via the processing order Oi. For example, when the processing order O3 is set to 1, the camera CAM corresponding to the processing order 1 is the camera CAM3, and the reading time R1n, the camera frame number G1n, and the drawing completion time F1n of the processing order 1 are stored in the camera CAM3. Corresponding information.

  The reading time Rjn indicates the reading time of the captured image CIMG read by the processor PU from the storage unit MEM in the display frame period Tn (the time when the processor PU reads the captured image CIMGi from the storage unit MEMi). For example, the reading time R1n indicates the reading time of the captured image CIMG that is first read from one of the storage units MEM1-MEM4 by the processor PU during the display frame period Tn. The reading time R2n indicates the reading time of the captured image CIMG that is read secondly from any of the storage units MEM1-MEM4 by the processor PU during the display frame period Tn.

  The camera frame number Gjn indicates the frame number of the captured image CIMG read by the processor PU from the storage unit MEM in the display frame period Tn. For example, the camera frame number G1n indicates the camera frame number of the captured image CIMG that is first read from any of the storage units MEM1-MEM4 by the processor PU during the display frame period Tn.

  The drawing completion time Fjn indicates the completion time of the drawing processing executed by the processor PU j-th during the display frame period Tn. For example, the drawing completion time F1n indicates the completion time of the drawing process executed first by the processor PU during the display frame period Tn. The drawing completion time F2n indicates the completion time of the drawing process executed second by the processor PU in the display frame period Tn.

  The frame drawing completion time Fn indicates the time when drawing processing is completed for all the captured images CIMG used for the display image DIMGn. That is, the frame drawing completion time Fn indicates the completion time of the drawing process last executed by the processor PU in the display frame period Tn (that is, the drawing completion time F4n).

  The display time Vn indicates the start time of the display frame period Tn. For example, the display time Vn indicates the time when the vertical synchronization signal is detected when the display image DIMG (n−1) is displayed on the display device DIS.

  FIG. 5 shows an example of the operation timing of the image processing system SYS shown in FIG. 5 indicate the capture time Cim, the read time Rjn, the drawing completion time Fjn, and the like recorded until the display frame period Tn ends. White diamonds indicate the capture time Ci (m + 1), the read time Rj (n + 1), and the like predicted in the display frame period Tn. In FIG. 5, the operation timing in the display frame period Tn will be mainly described.

  The processor PU follows the order set in the processing order O1, O2, O3, O4 (in the example shown in FIG. 5, O1 = 4, O2 = 3, O3 = 1, O4 = 2), and the captured images CIMG1-CIMG4 The drawing process for each of the above is executed. That is, in the display frame period Tn, the processor PU executes the drawing process in the order of the camera CAM3, the camera CAM4, the camera CAM2, and the camera CAM1.

  At the display time Vn, the captured image CIMG3m of the camera CAM3 is not stored in the storage unit MEM3. Therefore, the processor PU reads the captured image CIMG3 (m−1) from the storage unit MEM3 and executes a drawing process. The processor PU records the time when the captured image CIMG3 (m−1) is read from the storage unit MEM3 as the read time R1n. Then, when the drawing process for the photographed image CIMG3 (m−1) is completed, the processor PU records the completion time of the drawing process for the photographed image CIMG3 (m−1) as the drawing completion time F1n.

  The display delay time DLY3 of the photographed image CIMG3 is a time obtained by subtracting the capture time C3 (m−1) from the display time V (n + 1) because the drawing process has been performed on the photographed image CIMG3 (m−1). The display delay time DLY is a time DLY from when the captured image CIMG is stored in the storage unit MEM until it is displayed on the display device DIS. Hereinafter, the display delay time DLY is also referred to as a delay time DLY.

  When the drawing process for the photographed image CIMG3 (m−1) is completed, the processor PU reads the photographed image CIMG4m in the next processing order from the storage unit MEM4 and executes the drawing process. Note that at the drawing completion time F1n when the drawing process for the photographed image CIMG3 (m−1) is completed, the photographed image CIMG4m of the camera CAM4 is stored in the storage unit MEM4, and thus the photographed image CIMG4m is read from the storage unit MEM4. Therefore, the display delay time DLY4 of the captured image CIMG4 is a time obtained by subtracting the capture time C4m from the display time V (n + 1). The time when the captured image CIMG4m is read from the storage unit MEM4 is recorded as the reading time R2n.

  After the drawing process for the photographed image CIMG4m is completed, the drawing process for the photographed image CIMG2m is executed. After the drawing process for the photographed image CIMG2m is completed, the drawing process for the photographed image CIMG1m is executed. After the drawing process for the captured image CIMG1m is completed, the processing order Oi of the display frame period T (n + 1) next to the display frame period Tn is determined, and the display image DIMGn is displayed on the display device DIS at the display time V (n + 1). indicate.

In the process of determining the processing order Oi of the display frame period T (n + 1) next to the display frame period Tn, the optimal combination processing order is selected from all combinations of the processing order. In the image processing system SYS shown in FIG. 3, the number of camera CAMs is four, so there are 24 (= ₄ P ₄ = 4 × 3 × 2 × 1) combinations of processing orders. The white diamond shown in FIG. 5 shows one of 24 combinations (the same order as the display frame period Tn) as an example.

  When determining the processing order Oi of the display frame period T (n + 1) next to the display frame period Tn, the predicted value of the reading time R1 (n + 1) of the captured image CIMG read first in the display frame period T (n + 1) is displayed. Time V (n + 1) is assumed. The display time V (n + 1) is the start time of the display frame period T (n + 1). For example, the processor PU predicts, as the display time V (n + 1), a time obtained by adding a period of the display frame period T (a time obtained by subtracting the display time V (n−1) from the display time Vn) to the display time Vn.

  The predicted value of the reading time R2 (n + 1) of the captured image CIMG that is read secondly is obtained by adding the drawing processing time to the predicted value of the reading time R1 (n + 1) of the captured image CIMG that is read immediately before (that is, the first). It is time. In this case, the rendering process time is the rendering process time (= Fjn−Rjn) in the display frame period Tn of the captured image CIMG captured by the camera CAM that captured the captured image CIMG that is first read in the display frame period T (n + 1). ). In the example shown in FIG. 5, since the captured image CIMG3m captured by the camera CAM3 is read at the beginning of the display frame period T (n + 1), the processor PU in the display frame period Tn of the captured image CIMG3 captured by the camera CAM3. Refer to processing order O3. Then, since the processing order O3 is 1, the processor PU predicts the reading time R2 (n + 1) using the time (= F1n−R1n) obtained by subtracting the reading time R1n from the drawing completion time F1n as the drawing processing time.

  The predicted value of the read time Rj (n + 1) of the captured image CIMG read after the third is also calculated in the same manner as the predicted value of the read time R2 (n + 1) of the captured image CIMG read second.

  Further, the processor PU predicts the time (capture time Ci (m + 1) or the like) at which the captured image CIMG is updated for each camera CAM. For example, when the latest time when each captured image CIMGi is updated is the capture time Cim, the processor PU predicts the time obtained by adding the captured image update period Ui to the capture time Cim as the next update time. Furthermore, the processor PU predicts a time obtained by adding twice the captured image update period Ui to the capture time Cim as the next next update time. The captured image update period Ui is, for example, a time obtained by subtracting the capture time Ci (m−1) from the capture time Cim. Note that the processor PU may use a period assumed from the specification of the camera CAM as the captured image update period U. In the example shown in FIG. 5, the capture time Ci (m + 1) is calculated as the next update time.

  In addition, the processor PU calculates the camera frame number Gj (n + 1) of each captured image CIMGi that is to be rendered in the display frame period T (n + 1). Then, the processor PU calculates an evaluation value based on the camera frame number Gj (n + 1) or the like of each captured image CIMGi to be subjected to the drawing process in the display frame period T (n + 1). A method for calculating the camera frame number Gj (n + 1), a method for calculating the evaluation value, and the like will be described with reference to FIGS.

  The processor PU calculates evaluation values for all combinations of the processing order, and determines the processing order of the combination having the best evaluation value as the processing order Oi of the display frame period T (n + 1). Thereby, it is determined for each camera CAM which frame of each moving image photographed by each camera CAM is selected as the plurality of photographed images CIMG used for the display image DIMG. The example shown in FIG. 5 corresponds to the operation timing when the average value (for example, the mean square value) of the delay times DLY1, DLY2, DLY3, and DLY4 is used as the evaluation value.

  Here, for example, when the execution order of the drawing processing is fixed in the order of the camera CAM1, the camera CAM2, the camera CAM3, and the camera CAM4, the processor PU stores the captured image CIMG1 (m−1) at the display time Vn. Read from MEM3. That is, the processor PU performs a drawing process on the captured image CIMG1 (m−1) during the display frame period Tn. Then, a drawing process for the photographed image CIMG2 (m−1), a drawing process for the photographed image CIMG3 (m−1), and a drawing process for the photographed image CIMG4m are sequentially performed.

  As described above, when the processing order is fixed to the order based on the camera number i (for example, the order in which the camera number i is small), the cameras CAM1 and CAM2 capture the previous frame compared to the example shown in FIG. The image CIMG is used for the display image DIMGn. In this case, in the cameras CAM1 and CAM2, the delay time DLY is longer by the captured image update period U than the example shown in FIG. In other words, at the operation timing shown in FIG. 5, the delay times DLY1 and DLY2 are smaller than when the processing order is fixed in the order of the camera number i. That is, in the example illustrated in FIG. 5, the processor PU can reduce the average value of the delay times DLY1, DLY2, DLY3, and DLY4 as compared with the case where the processing order is fixed in the order of the camera number i.

  Here, for example, when a plurality of cameras CAM are operating with a clock system different from that of the processor PU, the captured image update period U may change in each of the plurality of cameras CAM depending on the accuracy of an oscillator or the like. is there. In this case, due to a change in the captured image update cycle U with the transition of time, there is a possibility that the transfer order of the plurality of captured images CIMG respectively captured by the plurality of cameras CAM to the storage unit MEM during the capturing is changed. Even in this case, since the shift of the captured image update period U occurs continuously, the measurement result of the operation timing of the display frame period Tn is used for prediction of the operation timing of the next display frame period T (n + 1). The influence of the change in the captured image update period U can be suppressed.

  As described above, the processor PU predicts the operation timing of the display frame period T (n + 1) based on the measurement result of the operation timing of the display frame period Tn, and executes the drawing processing execution order (that is, the captured image based on the prediction result). CIMG reading order) is determined. Thereby, as a plurality of captured images CIMG used for the display image DIMG, a captured image CIMGim having a camera frame number m suitable for conditions for improving the quality of the display image DIMG is determined for each camera CAM.

  FIG. 6 shows an example of the operation of the processor PU shown in FIG. The operation shown in FIG. 6 is realized by controlling the processor PU with software such as an image processing program. For example, the processor PU reads an image processing program and executes the operation shown in FIG. The program that causes the processor PU (ie, computer) to execute the operation illustrated in FIG. 6 is an aspect of the image processing program. When the functional blocks (functional blocks in the processor PU) corresponding to the capture time recording unit CREC shown in FIG. 3 are realized only by hardware, the operation shown in FIG. 6 is realized only by hardware. Also good.

  In FIG. 6, at the time when the process of step S380 is executed, the camera frame number of the latest captured image CIMGi among the captured images CIMGi for one frame stored in the storage unit MEMi is set as the camera frame number m as the processor PU. The operation of will be described. That is, the camera frame number m is the camera frame number of the captured image CIMGi used for the display image DIMGn. For example, in the relationship between the operation timing shown in FIG. 5 and the operation shown in FIG. 6, when the variable j shown in FIG. 6 is 1, the camera frame number m shown in FIG. m-1). When the variable j is 2, 3, or 4, the camera frame number m shown in FIG. 6 corresponds to the camera frame number m shown in FIG.

  In step S300, the processor PU sets a variable n indicating the display frame number n to 0, and moves the operation to step S320.

  In step S320, the processor PU records the display time Vn in a register or the like, and moves the operation to step S340. Note that before the first display image DIMGn (DIMG0) is displayed, the processor PU, for example, displays the start time of the first display frame period T0 of the display frame period Tn corresponding to the period of the vertical synchronization signal as the display time Vn ( V0) is recorded in a register or the like.

  In step S340, the processor PU sets a variable j indicating the processing order j to 1, and moves the operation to step S360.

  In step S360, the processor PU searches for the camera number i whose processing order Oi is the variable j, and moves the operation to step S380. Through the process in step S360, the camera CAMi that is the target of the drawing process is selected. Note that the processing order Oi is set to a predetermined default processing order (for example, in ascending order of the camera number i) when the processing in step S560 has never been executed.

  In step S380, the processor PU records the time when the m-th frame captured image CIMGim captured by the camera CAMi selected in the process of step S360 is stored in the storage unit MEMi as a capture time Cim in a register or the like. For example, the processor PU acquires a time stamp indicating the time when the captured image CIMGim is stored in the storage unit MEMi, and records the acquired time stamp as a capture time Cim in a register or the like. After the process of step S380 is executed, the operation of the processor PU moves to step S400.

  In step S400, the processor PU records the read time Rjn of the captured image CIMGim of the camera number i in a register or the like. For example, the processor PU reads the m-th frame captured image CIMGim captured by the camera CAMi selected in step S360 from the storage unit MEMi, and records the read time as a read time Rjn in a register or the like. After the process of step S400 is executed, the operation of the processor PU moves to step S420.

  In step S420, the processor PU performs a drawing process on the photographed image CIMGim of the camera number i (the photographed image CIMGim read from the storage unit MEMi in the process of step S400), and moves the operation to step S440.

  In step S440, the processor PU waits for execution of the process in step S460 until the drawing process for the captured image CIMGim is completed. That is, the processor PU executes the process of step S460 when the drawing process for the captured image CIMGim is completed.

  In step S460, the processor PU records the time when the drawing process on the captured image CIMGim is completed as a drawing completion time Fjn in a register or the like, and moves the operation to step S480.

  In step S480, the processor PU determines whether or not the variable j (processing order j) is 4. That is, the processor PU determines whether or not the drawing process has been completed for all the captured images CIMG1, CIMG2, CIMG3, and CIMG4 used for the display image DIMGn. For example, when the number of cameras CAM is N (N is a positive integer), the processor PU determines whether or not the variable j is N. When the variable j is 4, the operation of the processor PU moves to step S520. On the other hand, when the variable j is not 4, the operation of the processor PU moves to step S500.

  In step S500, the processor PU increments the variable j (j = j + 1), and moves the operation to step S360. That is, the processing in steps S360 to S460 is repeated according to the processing order set in the processing order Oi until drawing processing for one screen (drawing processing for all captured images CIMG used for the display image DIMGn) is completed. . When the drawing process for one screen is completed, the process of step S520 is executed.

  In step S520, the processor PU executes a flip process for switching the display buffer BUF. As a result, among the three buffers BUF in each storage unit MEM, the buffer BUF that stores the captured image CIMGim that has been subjected to the drawing process in the process of step S420 is switched to the display buffer BUF. After the process of step S520 is executed, the operation of the processor PU moves to step S540.

  In step S540, the processor PU executes order determination processing to determine the next processing order. For example, the processor PU executes the order determination process shown in FIG. Or processor PU performs the order determination process shown in FIG. By executing the order determination process, the order of the drawing process executed in the display frame period T (n + 1) next to the current display frame period Tn is determined. After the process of step S540 is executed (after the next process order is determined by the order determination process), the operation of the processor PU moves to step S560.

  In step S560, the processor PU sets the processing order determined by the order determination processing in step S540 as the processing order Oi. As a result, the processing order Oi is updated to the processing order determined by the order determination processing in step S540. After the process of step S560 is executed, the operation of the processor PU moves to step S580.

  In step S580, the processor PU waits for execution of the process of step S600 until the display image DIMGn is displayed on the display device DIS. For example, the processor PU waits for execution of the process of step S600 until it detects a vertical synchronization signal or the like. That is, the processor PU executes the process of step S600 in synchronization with the timing (for example, a vertical synchronization signal) for displaying the display image DIMGn on the display device DIS.

  In step S600, the processor PU displays a plurality of captured images CIMG1, CIMG2, CIMG3, and CIMG4 on which the drawing process has been completed on the display device DIS as one display image DIMGn. For example, the processor PU transfers the captured images CIMG1, CIMG2, CIMG3, and CIMG4 stored in the display buffer BUF in each storage unit MEM to the display device DIS as one display image DIMGn. After the process of step S600 is executed, the operation of the processor PU moves to step S620.

  In step S620, the processor PU increments the variable n (n = n + 1), and moves the operation to step S320. Thereby, the process of displaying the next display image DIMGn on the display device DIS is executed. By repeating the processes of steps S320 to S620, the display image DIMG is sequentially displayed on the display device DIS in synchronization with the vertical synchronization signal. The operation of the processor PU is not limited to the example shown in FIG.

  FIG. 7 shows an example of the order determination process (step S540) shown in FIG. FIG. 7 shows an example of the order determination process when the time from when the captured image CIMGi is stored in the storage unit MEMi until it is displayed on the display device DIS is set as the evaluation value Ek. The camera frame number m shown in FIG. 7 is the camera frame number of the captured image CIMGi used for the display image DIMGn, as in FIG.

  In step S541, the processor PU predicts the display times V (n + 1) and V (n + 2), and sets a variable k indicating a combination pattern of processing orders to 1. Hereinafter, a combination of patterns specified by the variable k is also referred to as a combination k.

  As described with reference to FIG. 5, the display time V (n + 1) corresponds to the start time of the display frame period T (n + 1) next to the current display frame period Tn. For example, the processor PU predicts, as the display time V (n + 1), a time obtained by adding a period of the display frame period T (a time obtained by subtracting the display time V (n−1) from the display time Vn) to the display time Vn. The display time V (n + 2) corresponds to the start time of the display frame period T (n + 2) next to the current display frame period Tn. For example, the processor PU predicts a time obtained by adding twice the period of the display frame period T to the display time Vn as the display time V (n + 2).

  Next, in step S542, the processor PU selects the processing order combination k, sets a variable j indicating the processing order j to 1, and moves the operation to step S543.

  In step S543, the processor PU selects the camera CAMi of the processing order j, and moves the operation to step S544. In each combination k, the camera CAMi of the processing order j is uniquely determined with respect to the processing order j.

  For example, when the processing order indicated by the combination k is the order of the camera CAM3, the camera CAM4, the camera CAM2, and the camera CAM1, the camera CAMi of the processing order 1 is the camera CAM3. The cameras CAMi of the processing order 2, the processing order 3, and the processing order 4 are the cameras CAM4, CAM2, and CAM1, respectively. Alternatively, when the processing order indicated by the combination k is the order of the camera CAM1, the camera CAM3, the camera CAM4, and the camera CAM2, the camera CAMi in the processing order 1 is the camera CAM1. The cameras CAMi in the processing order 2, the processing order 3, and the processing order 4 are the cameras CAM3, CAM4, and CAM2, respectively.

  In step S544, the processor PU predicts the reading time Rj (n + 1), the capture time Ci (m + 1), and Ci (m + 2) as described with reference to FIG. After the process of step S544 is executed, the operation of the processor PU moves to step S545.

  In step S545, the processor PU calculates the camera frame number Gj (n + 1) based on the relationship between the read time Rj (n + 1) predicted in the process of step S544 and the capture times Ci (m + 1) and Ci (m + 2). .

  For example, when the read time Rj (n + 1) is a time before the capture time Ci (m + 1), the camera frame number Gj (n + 1) is set to “m” (Gj (n + 1) = m). In this case, for example, the captured image CIMGim having the same frame is used for the display image DIMGn and the display image DIMG (n + 1).

  For example, when the read time Rj (n + 1) is a time after the capture time Ci (m + 1) and before the capture time Ci (m + 2), the camera frame number Gj (n + 1) is “m + 1”. (Gj (n + 1) = m + 1). In this case, for example, the captured image CIMGi (m + 1) of the next frame of the captured image CIMGim is used as the display image DIMG (n + 1).

  For example, when the read time Rj (n + 1) is a time after the capture time Ci (m + 2), the camera frame number Gj (n + 1) is set to “m + 2” (Gj (n + 1) = m + 2). In this case, for example, the captured image CIMGi (m + 2) of the next frame after the captured image CIMGim is used as the display image DIMG (n + 1). That is, the captured image CIMGi is skipped by one frame.

  After the process of step S545 is executed, the operation of the processor PU moves to step S546.

  In step S546, the processor PU calculates the delay time DLYi based on the camera frame number Gj (n + 1) calculated in the process of step S545, the display time V (n + 2) predicted in the process of step S541, and the like.

  When the camera frame number Gj (n + 1) is “m”, the delay time DLYi is a time obtained by subtracting the capture time Cim recorded in the process of step S380 shown in FIG. 6 from the display time V (n + 2) (DLYi). = V (n + 2) -Cim).

  When the camera frame number Gj (n + 1) is “m + 1”, the delay time DLYi is a time obtained by subtracting the capture time Ci (m + 1) predicted in the process of step S544 from the display time V (n + 2) (DLYi = V (N + 2) -Ci (m + 1)).

  When the camera frame number Gj (n + 1) is “m + 2”, the delay time DLYi is a time obtained by subtracting the capture time Ci (m + 2) predicted in the process of step S544 from the display time V (n + 2) (DLYi = V (N + 2) -Ci (m + 2)).

  After the process of step S546 is executed, the operation of the processor PU moves to step S547.

  In step S547, the processor PU determines whether or not the variable j (processing order j) is 4. That is, the processor PU determines whether or not the delay times DLY1, DLY2, DLY3, and DLY4 of the captured images CIMG1, CIMG2, CIMG3, and CIMG4 have been calculated. For example, when the number of cameras CAM is N (N is a positive integer), the processor PU determines whether or not the variable j is N. If the variable j is 4, the operation of the processor PU moves to step S549. On the other hand, when the variable j is not 4, the operation of the processor PU moves to step S548.

  In step S548, the processor PU increments the variable j (j = j + 1), and moves the operation to step S543. That is, the processes in steps S543 to S546 are repeated until the delay times DLY1, DLY2, DLY3, and DLY4 for the combination k are calculated. When the delay times DLY1, DLY2, DLY3, and DLY4 for the combination k are calculated, the process of step S549 is executed.

  In step S549, the processor PU calculates an evaluation value Ek using the delay time DLYi calculated in the process of step S546, and moves the operation to step S550. The evaluation value Ek is expressed by Expression (1) using the number N of cameras CAM (N = 4 in the example shown in FIG. 7) and the delay time DLYi.

  As described above, in the example illustrated in FIG. 7, the processor PU calculates the mean square of the delay time DLYi as the evaluation value Ek.

  In step S550, the processor PU determines whether or not the variable k indicating the combination pattern of the processing order is 24. That is, the processor PU determines whether or not the evaluation value Ek has been calculated for all combinations k of the processing order of the captured images CIMG1, CIMG2, CIMG3, and CIMG4. For example, when the number of cameras CAM is N (N is a positive integer), the processor PU determines whether or not the variable k is a factorial of N. When the variable k is 24, the operation of the processor PU moves to step S552. On the other hand, when the variable k is not 24, the operation of the processor PU moves to step S551.

  In step S551, the processor PU increments the variable k (k = k + 1), and moves the operation to step S542. That is, the processes in steps S542 to S549 are repeated until the evaluation value Ek is calculated for all combinations k of the processing order of the captured images CIMG1, CIMG2, CIMG3, and CIMG4. When the evaluation value Ek is calculated for all the combinations k, the process of step S552 is executed.

  In step S552, the processor PU determines the processing order of the combination k that minimizes the evaluation value Ek calculated in the processing of step S549 as the next processing order. That is, the processor PU is a combination that minimizes the evaluation value Ek related to the display delay time DLY (in the example illustrated in FIG. 7, the mean square of the delay times DLY1, DLY2, DLY3, and DLY4) among all the combinations k in the processing order. The processing order of k is determined as the next processing order.

  When the process of step S552 ends, the order determination process ends, and the process order Oi is updated in step S560 shown in FIG. For example, the processor PU updates the processing order Oi to the processing order determined in the processing of step S552 in step S560 illustrated in FIG. Note that the evaluation value Ek related to the display delay time DLY is not limited to the example shown in FIG. 7 (root mean square of the delay time DLYi).

  FIG. 8 shows another example of the order determination process (step S540) shown in FIG. FIG. 8 shows an example of the order determination process when the variation of the capture times Cim of the captured images CIMG1m, CIMG2m, CIMG3m, and CIMG4m used for the display image DIMGn is set as the evaluation value Ek. The camera frame number m shown in FIG. 8 is the camera frame number of the captured image CIMGi used for the display image DIMGn, as in FIG. Steps that are the same as or similar to the steps described in FIG. 7 are given the same or similar reference numerals, and detailed descriptions thereof are omitted.

  Steps S541-S545 are the same as or similar to steps S541-S545 shown in FIG. After the process of step S545 is executed, the operation of the processor PU moves to step S546a.

  In step S546a, the processor PU calculates the capture time Ci based on the camera frame number Gj (n + 1) calculated in the process of step S545.

  When the camera frame number Gj (n + 1) is “m”, the capture time Ci is the time when the captured image CIMGim is stored in the storage unit MEMi (Ci = Cim). That is, the capture time Ci is the capture time Cim recorded in the process of step S380 shown in FIG.

  When the camera frame number Gj (n + 1) is “m + 1”, the capture time Ci is the capture time Ci (m + 1) predicted in the process of step S544 (Ci = Ci (m + 1)).

  When the camera frame number Gj (n + 1) is “m + 2”, the capture time Ci is the capture time Ci (m + 2) predicted in the process of step S544 (Ci = Ci (m + 2)).

  After the process of step S546a is executed, the operation of the processor PU moves to step S547.

  Steps S547 and S548 are the same as or similar to steps S547 and S548 shown in FIG. For example, in step S547, the processor PU determines whether or not the variable j (processing order j) is 4. If the variable j is 4, the processor PU moves the operation to step S549a. On the other hand, if the variable j is not 4, the processor PU increments the variable j (j = j + 1) in step S548, and moves the operation to step S543.

  In step S549a, the processor PU calculates an evaluation value Ek using the capture time Ci calculated in the process of step S546a, and moves the operation to step S550. The evaluation value Ek is expressed by Expression (2) using the number N of cameras CAM (N = 4 in the example shown in FIG. 7) and the capture time Ci.

  A symbol AVEc in Expression (2) indicates an average of N capture times Ci. For example, when the number N of the cameras CAM is 4, the average AVEc is expressed by Expression (3) using the capture times Ci (capture times C1, C2, C3, and C4) calculated in the process of step S546a. The

AVEc = (C1 + C2 + C3 + C4) / 4 (3)
As described above, in the example illustrated in FIG. 8, the processor PU calculates the variance of the capture time Ci as the evaluation value Ek.

  Steps S550-S552 are the same as or similar to steps S550-S552 shown in FIG. For example, in step S550, the processor PU determines whether or not the variable k is 24. If the variable k is 24, the processor PU moves the operation to step S552. On the other hand, when the variable k is not 24, the processor PU increments the variable k (k = k + 1) in step S551, and moves the operation to step S542.

  In step S552, the processor PU determines the processing order of the combination k that minimizes the evaluation value Ek calculated in step S549a as the next processing order. That is, the processor PU has the smallest evaluation value Ek (variation of capture times C1, C2, C3, and C4 in the example shown in FIG. 7) regarding the variation in the capture time Cim among all the combinations k in the processing order. The processing order of k is determined as the next processing order.

  When the process of step S552 ends, the order determination process ends, and the process order Oi is updated in step S560 shown in FIG. For example, the processor PU updates the processing order Oi to the processing order determined in the processing of step S552 in step S560 illustrated in FIG.

  Note that the order determination process (step S540) is not limited to the example shown in FIG. 7 or FIG. For example, when the processor PU detects a combination k that has an evaluation value Ek that is equal to or less than a predetermined target value even before the calculation of the evaluation value Ek ends for all the combinations k, the processor PU performs the process of step S552. Also good. In this case, the processor PU determines the processing order of the combination k that becomes the evaluation value Ek equal to or lower than the target value as the next processing order.

  Further, both the variance of the capture time Cim and the root mean square of the delay time DLYi may be used as the evaluation value Ek. For example, the processor PU may determine the processing order of the combination k that minimizes the mean square of the delay time DLYi among the combinations k in which the variance of the capture time Ci is equal to or less than the target value, as the next processing order. Alternatively, the processor PU may determine, as the next processing order, the processing order of the combination k that minimizes the variance of the capture time Ci among the combinations k in which the root mean square of the delay times DLYi is equal to or less than the target value.

  FIG. 9 shows an example of a simulation result when the mean square of the display delay time DLYi is the evaluation value Ek. The horizontal axis of FIG. 9 shows the mean square of the display delay time DLYi. In addition, the vertical axis in FIG. 9 indicates the frequency of each section when the mean square of the display delay time DLYi is divided at 0.1 intervals. In the example illustrated in FIG. 9, the simulation is executed 100 times while randomly changing the input timing of the captured image CIMGi from the camera CAMi to the storage unit MEMi. The simulation conditions are shown below. The number of cameras CAMi is 4, the captured image update period Ui of each camera CAMi is 100, and the display frame period T is 100.

  The shaded rectangle shown in FIG. 9 indicates the simulation result when the execution order of the drawing process is adjusted (after optimization) based on the order determination process shown in FIG. In addition, a white rectangle illustrated in FIG. 9 indicates a simulation result when the execution order of the drawing processing is fixed (before optimization).

  When the execution order of the drawing process is adjusted, the average and variation of 100 times of the square average of the display delay time DLYi are smaller than when the execution order of the drawing process is fixed. For example, when the execution order of the drawing processing is adjusted, the average of 100 simulations of the square average of the display delay time DLYi is 0.91. On the other hand, when the execution order of the drawing processing is fixed, the average of 100 simulations of the square average of the display delay time DLYi is 1.48.

  As described above, by adjusting the execution order of the drawing process based on the order determination process shown in FIG. 7, the plurality of captured images CIMGi in the display image DIMGi are compared with the case where the execution order of the drawing process is fixed. The mean square of the display delay time DLYi can be reduced.

  FIG. 10 shows an example of a simulation result when the variance of the capture time Ci is the evaluation value Ek. The horizontal axis of FIG. 10 shows the variance of the capture time Ci. The vertical axis in FIG. 10 indicates the frequency of each section when the variance of the capture time Ci is divided at 50 intervals. Also in the example illustrated in FIG. 10, similarly to FIG. 9, the simulation is executed 100 times by randomly changing the input timing of the captured image CIMGi from the camera CAMi to the storage unit MEMi.

  The simulation conditions and the meaning of the white rectangle shown in FIG. 10 are the same as those in FIG. Note that the shaded rectangle shown in FIG. 10 indicates the simulation result when the execution order of the drawing process is adjusted (after optimization) based on the order determination process shown in FIG.

  When the execution order of the drawing process is adjusted, the average and variation of the 100 simulations of the dispersion of the capture time Ci are smaller than when the execution order of the drawing process is fixed. For example, when the execution order of the drawing processing is adjusted, the average of 100 simulations of the dispersion of the capture time Ci is 186. On the other hand, when the execution order of the drawing processing is fixed, the average of 100 simulations of the dispersion of the capture time Ci is 474.

  In this manner, by adjusting the execution order of the drawing process based on the order determination process shown in FIG. 8, compared with the case where the execution order of the drawing process is fixed, in the plurality of captured images CIMGi in the display image DIMGi. Variations in the capture time Ci can be reduced.

  FIG. 11 shows an example of the hardware configuration of the image processing system SYS shown in FIG. The image processing system SYS is realized by a computer device CP, a plurality of cameras CAM (CAM1, CAM2, CAM3, CAM4) connected to the computer device CP, and a display device DIS connected to the computer device CP.

  The computer device CP includes a processor PU, memories MMEM, VMEM, a hard disk device HDD, an input / output interface IF, and an optical drive device ODR. The processor PU, the memories MMEM, VMEM, the hard disk device HDD, the input / output interface IF, and the optical drive device ODR are connected to each other via a bus BUS.

  The optical drive device ODR can be mounted with a removable disk RD such as an optical disk, and reads and records information recorded on the mounted removable disk RD.

  The input / output interface IF is connected to, for example, the cameras CAM1, CAM2, CAM3, and CAM4. Accordingly, the processor PU and the like can receive the captured images CIMG1, CIMG2, CIMG3, and CIMG4 captured by the cameras CAM1, CAM2, CAM3, and CAM4 via the input / output interface IF.

  For example, the memory VMEM stores captured images CIMG1, CIMG2, CIMG3, and CIMG4 captured by the cameras CAM1, CAM2, CAM3, and CAM4. That is, the storage units MEM1, MEM2, MEM3, and MEM4 illustrated in FIG. 3 are allocated to the memory VMEM.

  The memory MMEM stores, for example, the operating system of the computer apparatus CP. Further, the memory MMEM stores an application program such as an image processing program for the processor PU to execute the operation of the image processing apparatus. An application program such as an image processing program can be recorded and distributed on a removable disk RD such as an optical disk.

  For example, the computer apparatus CP may read an application program such as an image processing program from the removable disk RD via the optical drive apparatus ODR and store it in the memory MMEM and the hard disk apparatus HDD. Further, the computer device CP may download an application program such as an image processing program via a communication device (not shown) connected to a network such as the Internet and store it in the memory MMEM and the hard disk device HDD.

  The processor PU executes the image processing program stored in the memory MMEM or the like, thereby realizing the functions of functional blocks such as the capture time recording unit CREC in the processor PU shown in FIG.

  Note that the hardware configuration of the image processing system SYS is not limited to the example shown in FIG. For example, the computer apparatus CP may have an interface for a storage medium such as a USB (Universal Serial Bus) memory. Alternatively, the optical drive device ODR may be omitted from the computer device CP. Further, for example, the memory VMEM may be omitted from the computer device CP, and the storage units MEM1-MEM4 illustrated in FIG. 3 may be allocated to the memory MMEM.

  Further, the processor PU, the memory MMEM, the VMEM, the hard disk device HDD, and the input / output interface IF may realize the functions of the image processing device 20 illustrated in FIG. For example, the processor PU may implement the functions of the image processing apparatus 20 illustrated in FIG. 1 by executing an image processing program stored in the memory MMEM or the like. In this case, the storage unit 30 illustrated in FIG. 1 may be allocated to the memory VMEM or the memory MMEM.

  As described above, also in the embodiment shown in FIGS. 3 to 11, the same effect as that of the embodiment shown in FIGS. 1 and 2 can be obtained. In the embodiment shown in FIGS. 3 to 11, the processor PU calculates the evaluation value Ek based on the display time Vn, the capture time Cim, the reading time Rjn, and the drawing completion time Fjn, and draws based on the calculated evaluation value Ek. Determine the processing execution order.

  For example, when the evaluation value Ek is the mean square of the display delay time DLYi, the execution order of the drawing processing is set so that the mean square of the display delay times DLYi of the plurality of captured images CIMGi in the next display image DIMG is as small as possible. Can be determined. Further, for example, when the evaluation value Ek is a variance of the capture time Ci, the execution order of the drawing processing is determined so that the shift of the capture time Ci of the plurality of captured images CIMGi in the next display image DIMG is as small as possible. be able to. Thus, the quality of the display image DIMG can be improved according to the type of the evaluation value Ek.

  From the above detailed description, features and advantages of the embodiments will become apparent. This is intended to cover the features and advantages of the embodiments described above without departing from the spirit and scope of the claims. Also, any improvement and modification should be readily conceivable by those having ordinary knowledge in the art. Therefore, there is no intention to limit the scope of the inventive embodiments to those described above, and appropriate modifications and equivalents included in the scope disclosed in the embodiments can be used.

  DESCRIPTION OF SYMBOLS 10 ... Image processing system; 20 ... Image processing apparatus; 30 ... Memory | storage part; 40 ... Drawing process part; 42 ... Capture time recording part; 44 ... Drawing start time recording part; 50: Imaging device; 60: Display device; CAL ... Prediction unit; CAM ... Camera; CP ... Computer device; CREC ... Capture time recording unit; DIS ... Display device; DRAW ... Drawing processing unit; FREC ... Drawing completion time recording HDD: Hard disk device; IF: I / O interface; MEM: Memory unit: MMEM, VMEM: Memory; ODR: Optical drive device; PU: Processor; RD: Removable disk; RREC: Reading time recording unit: SEQ: Order determination Section: SYS Image processing system VREC Display time recording section

Claims (5)

With reference to a storage unit that stores a plurality of captured images respectively captured by a plurality of imaging devices for each frame of each imaging device, the plurality of captured images are sequentially displayed as display images on a display device at predetermined time intervals. In the image processing method,
A capture time indicating a time when the plurality of captured images are stored in the storage unit is recorded in association with each of the plurality of imaging devices,
For each of the plurality of captured images stored in the storage unit, a drawing process for displaying on the display device is sequentially performed,
A drawing start time indicating a time at which the drawing process is started is recorded in association with each of the plurality of imaging devices,
A drawing completion time indicating a time when the drawing process is completed is recorded in association with each of the plurality of imaging devices,
The execution order of the drawing process for each of the plurality of captured images when the next display image is displayed on the display device is determined based on the already recorded capture time, the drawing start time, and the drawing completion time. An image processing method characterized by: The image processing method according to claim 1,
Record the display time indicating the time when the display image was displayed on the display device,
Using the display time, the capture time, the drawing start time, and the drawing completion time that have already been recorded, the capture time, the drawing start time, and the drawing completion time corresponding to the next display image are set to the captured images. Predict for each combination of processing order,
An image processing method, comprising: determining an execution order of the drawing process when displaying a next display image on the display device based on the predicted capture time, drawing start time, and drawing completion time. The image processing method according to claim 2,
Based on the predicted capture time, drawing start time, and drawing completion time, an evaluation value related to the time from when the captured image is stored in the storage unit until it is displayed on the display device is calculated for each combination.
An image processing method, comprising: determining an execution order of the drawing processing when displaying a next display image on the display device based on the evaluation value. The image processing method according to claim 2,
Based on the predicted capture time, drawing start time, and drawing completion time, an evaluation value related to variation in capture time is calculated for each combination,
An image processing method, comprising: determining an execution order of the drawing processing when displaying a next display image on the display device based on the evaluation value. With reference to a storage unit that stores a plurality of captured images respectively captured by a plurality of imaging devices for each frame of each imaging device, the plurality of captured images are sequentially displayed as display images on a display device at predetermined time intervals. In an image processing program for causing a computer to execute image processing,
A capture time indicating a time when the plurality of captured images are stored in the storage unit is recorded in association with each of the plurality of imaging devices,
For each of the plurality of captured images stored in the storage unit, a drawing process for displaying on the display device is sequentially performed,
A drawing start time indicating a time at which the drawing process is started is recorded in association with each of the plurality of imaging devices,
A drawing completion time indicating a time when the drawing process is completed is recorded in association with each of the plurality of imaging devices,
The execution order of the drawing process for each of the plurality of captured images when the next display image is displayed on the display device is determined based on the already recorded capture time, the drawing start time, and the drawing completion time. An image processing program characterized by:

Images