JP2021114754A - Image processing system - Google Patents

Abstract

To provide an image processing system for distributing a moving image from a server to a client, where a RAW moving image in the server is enabled to be appropriately developed and transmitted.SOLUTION: An image processing system includes a server for performing development processing on a RAW moving image and a client for performing parameter setting and reproduction operation on the RAW moving image, the server connected to the client via a communication network. The server comprises: reception means for receiving a development parameter and reproduction information from the client; development processing means for performing development processing on the RAW moving image according to the development parameter; first control means for controlling the development processing on the basis of the reproduction information; second control means for controlling and switching a transmission method on the basis of the reproduction information; and transmission means for transmitting a moving image obtained by the development processing to the client. The client comprises development parameter setting means, reproduction operation means, transmission means for transmitting the development parameter and the reproduction information to the server, and reception means for receiving the moving image from the server.SELECTED DRAWING: Figure 3

Description

The present invention relates to a technique for developing a RAW moving image according to the reproduction information received from the client and transmitting the RAW moving image to the client.

As a method for playing back the video recorded on the server, a method of streaming the video from the server to the client is generally used. In such streaming distribution, the server stores in advance stream data encoded for a plurality of transmission speeds so that the moving image can be distributed with appropriate image quality in various transmission environments.

The server described in Patent Document 1 calculates the transmission speed, selects stream data according to the transmission speed, and transmits the stream data to the client.

Japanese Unexamined Patent Publication No. 2003-169091

The constitution of Patent Document 1 is as follows. It is effective when handling general moving image data encoded by 264 or HEVC. However, when the server device handles RAW moving images, there is a problem that it is difficult to utilize the characteristics of RAW moving images.

The RAW moving image is a moving image in which a RAW image is recorded as each frame data of the moving image, and the RAW image is a data in which the RAW data obtained from the image sensor of the Bayer array is recorded. In order to obtain reproducible image data from a RAW image, it is necessary to develop the RAW image, and the hue, brightness, white balance, etc. of the image data vary depending on the values of the parameters (development parameters) used for the development processing of the RAW image. It is decided.

The user adjusts the setting of the development parameter value to adjust the hue and brightness that he / she desires.

Therefore, when the server handles RAW video, the image data itself to be displayed changes depending on the setting value of the RAW video development parameter. Therefore, if the configuration is such that a pre-encoded video is prepared, the video that reflects the user's RAW development setting can be displayed. There was a problem that it could not be confirmed. Further, when it is desired to confirm the RAW development setting with a high-quality display, there is a problem that the image quality cannot be satisfied depending on the transmission speed.

One embodiment of the present invention is an image processing system in which a server that develops RAW moving images and a client that sets parameters and plays back RAW moving images are connected by a communication network. A receiving means for receiving reproduction information, a development processing means for developing a RAW moving image according to development parameters, a first control means for controlling development processing with reproduction information, and a second control for switching which controls a transmission method with reproduction information. A means and a transmission means for transmitting a moving image obtained by development processing to a client, the client includes a development parameter setting means, a reproduction operation means, and a transmission means for transmitting development parameters and reproduction information to a server. An image processing system including a receiving means for receiving a moving image from a server.

According to the present invention, it is possible to appropriately develop and transmit a RAW moving image on a server.

It is a block diagram which shows the structure of the client of the RAW moving image development system which concerns on this embodiment. It is a figure which showed the outline of the whole of the RAW moving image development system which concerns on this embodiment. It is a flow chart which shows the RAW moving image processing procedure which concerns on this Embodiment. It is a flow chart which shows the RAW development processing procedure which concerns on this Embodiment. It is a flow chart which shows the determination processing procedure of the reproduction method and development processing which concerns on this Embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The embodiment described below is an example as a means for realizing the present invention, and may be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions.

This embodiment is a RAW moving image development system (image processing system) in which a server for developing RAW moving images and a client for setting parameters and playing back RAW moving images are connected by a communication network.

The client operates according to an application program in which the user performs operations related to RAW video. The server performs RAW development processing in response to an instruction from the client, and operates according to a program that transmits the RAW development result to the client.

The RAW moving image handled in this embodiment has a color depth of 12 bits, and is a RAW moving image in which a RAW image is recorded as uncompressed as each frame data of the moving image. The present invention can be applied to other forms of RAW moving images such as compressed RAW moving images.

(Explanation of the entire system)
FIG. 2 is a diagram showing an outline of the entire RAW moving image development system of the present embodiment.
As shown in FIG. 2, the RAW video development system includes a client 210, an Internet 220 as a communication network, a server 230, and a storage 240. The client 210 and the server 230 are connected via the Internet 220. Further, the server 230 is connected to the storage 240. As shown in FIG. 2, the storage 240 may be a storage provided outside the server 230 or a storage provided inside the server 230.

The client 210 stores a RAW moving image processing program that sets development parameters and reproduces RAW moving images.

The server 230 stores a RAW development processing program that develops a RAW moving image.

There are at least two types of development processing that can be performed by the RAW development processing program, high-quality development processing and high-speed development processing, and the high-quality development processing performs development processing with 12 bits. The high-speed development process is an 8-bit development process. Further, in the high-speed development processing, the noise reduction processing is executed by the simplified version, so that the development can be performed at a higher speed than the high-quality development processing. That is, conversely, the high-quality development process is a low-speed development process as compared with the high-speed development process, and the high-speed development process is a low-quality development process as compared with the high-quality development process.

A RAW moving image file is stored in the storage 240.

Details of each program will be described later.

(Client description)
FIG. 1 is a block diagram showing a configuration of a client 210. The client 210 is realized by a PC and a program running on the PC.

In FIG. 1, 190 is the client 210. Reference numeral 101 denotes a control unit that controls the entire client 210, for example, a Central Processing Unit (CPU). Reference numeral 102 denotes a Read Only Memory (ROM) for storing programs and parameters that do not need to be changed. Reference numeral 103 denotes a Random Access Memory (RAM) that temporarily stores programs and data supplied from an external device or the like.

The 104 is an external hard disk or memory card fixedly installed on the client 210, an optical disk such as a floppy disk (FD) or a compact disk (CD) that can be attached to and detached from the computer device 100, a magnetic or optical card, an IC card, or the like. It is a storage device. The external storage device 104 can record digital data such as programs and image files. A RAW moving image processing program, which will be described later, is recorded in the external storage device 104 as a readable program code in the control unit 101, and the control unit 101 executes the program code.

Reference numeral 105 denotes an interface with an input device such as a pointing device (mouse) or a keyboard 109 that receives a user's operation and inputs data. Reference numeral 106 denotes an interface with the display 110 for displaying the data held by the computer device 100 and the supplied data.

Reference numeral 107 denotes a network interface for connecting to a network line 113 such as the Internet. It is connected to the server 230 through a network line.

Reference numeral 108 denotes a system bus for communicably connecting the units 101 to 107.

In the present embodiment, an example in which the RAW moving image processing program is recorded in the external storage device 104 is shown, but the present invention can be applied even when the program is on the ROM 102 or the network.

Next, the operation of the client according to the present embodiment will be described.

FIG. 3 is a flowchart showing a procedure of a RAW moving image processing program executed by the control unit 101 in FIG.

In step S300, the control unit 101 displays the initial screen of the RAW moving image processing program. Similar to a general application, it is a process of displaying a window (not shown).

The UI includes an end button, a file specification button, a playback control, a display size setting control, and a development parameter setting control. If no file is selected, only the end button and file specification button can be operated. Immediately after starting the program, the file is not selected.

The control unit 101 measures the communication speed with the server 230 in step S302. The measurement method may be a general method.

In step S305, the control unit 101 makes a file list request to the server 230, and acquires a readable RAW video file list from the server 230 (file list request to the server). At this time, the client 210 and the server 230 are connected via the Internet.

In step S310, the control unit 101 specifies a RAW moving image file to be developed. The control unit 101 displays a file designation control (not shown) on the display 110. The RAW video file list acquired in step S305 is displayed on the file designation controller. This allows the user to select the RAW moving image file in the storage 240. The file setting control is configured to use, for example, the file open dialog provided as standard in the OS (Operating System), and is configured to allow one image file to be selected.

Further, in step S310, the control unit 101 transmits the file name of the file selected by the user as the file name of the RAW moving image file targeted by the server (file name is set in the server).

Further, in step S310, the control unit 101 sets all the buttons in the selectable state after the file name designation process.

In step S315, the control unit 101 is a process of setting the development parameter value according to the user operation of the development parameter setting control (not shown). The development parameters that can be set in this embodiment are the settings of white balance, brightness, contrast, and noise reduction. Of course, other settings may be possible. If the user does not operate anything, this step moves to the next step without processing anything.

In step S320, the control unit 101 is a process of setting the reproduction state according to the user operation of the reproduction control (not shown). Playback operations that can be operated by the user include play (playback) and pause (stop) of the moving image. An interface such as a playback slider can be applied as a playback control.

When the user operates the playback control to perform a play operation, the playback status information is set to the play status. When the user operates the playback control to perform a pause operation, the playback status information is set to the pause state.

If the user does not operate anything, this step does not process anything, maintains the playback state information, and moves to the next step.

In step S325, the control unit 101 is a process of setting display size information according to a user operation of the display size setting control (not shown). The display size that can be selected by the user is 100% display, 50% display, and 25% display. Of course, other display sizes may be selectable.

If the user does not operate anything, this step does not process anything, maintains the display size information, and moves to the next step.

In step S330, the control unit 101 performs a process of creating reproduction information. The playback information is composed of playback status information, transmitted moving image information, and communication speed information. Here, the playback state information is information indicating the playback state of the moving image on the client, the transmitted moving image information is information indicating the transmission form of the moving image to the client, and the communication speed information is information indicating the communication speed in the communication network. The reproduction information may include other information.

In step S330, the control unit 101 sets the reproduction state information set in step 320 as the reproduction information. Further, in step S330, the control unit 101 sets the display size information set in step S325 as the transmitted moving image information. For the communication speed information, the communication speed information measured in step S302 is set.

In the present embodiment, an example of setting the display size information as the transmitted moving image information is shown, but other configurations may be used. For example, the configuration may be such that the bit rate of the moving image and the image quality (quality) of the moving image are set instead of the image size.

In step S335, the control unit 101 performs a process of transmitting development parameters and reproduction information to the server, respectively.

In step S340, the control unit 101 requests the server for image data and performs a process of receiving the developed image data from the server.

At this time, information (transmission method information) indicating whether the image data is transmitted from the server as a streaming format or as one frame of image data is also acquired.

In step S345, the control unit 101 performs a process of displaying the image data acquired in step S340.

When it is received by streaming, processing related to streaming is performed and displayed. When the image data of one frame is acquired, the process of displaying the image data is performed.

In step S350, the control unit 101 determines whether or not the file specification button has been pressed by the user. If the button is pressed, the process proceeds to step S355. If the button is not pressed, the process proceeds to step S360.

In step S355, the process shifts when the file specification button is pressed. In step S355, the control unit 101 sets the reproduction state information of the reproduction information to the pause state and transmits the reproduction information to the server in order to put the moving image reproduction into the pause state. After that, the process is moved to step S302 so that the process of specifying the file can be started again.

In step S360, the control unit 101 determines whether or not the end button has been pressed by the user. If the button is pressed, exit the program. If the button is not pressed, the process is moved to step S315, and the reproduction process is repeated.

By repeating steps S315 to S360, the image data reflecting the development parameter settings can be reproduced and displayed on the client side. It should be noted that one process from step S315 to step S345 is an update of the image display of one frame, and by repeating this process many times, the image is updated and displayed as a moving image.

(Description of server)
The RAW development server of this embodiment uses a workstation. Since the configuration of the server is the same as that of the client, it will be described with reference to FIG. The server 230 of the present embodiment is realized by a workstation and a program running on the workstation.

In FIG. 1, 190 is the server 230. Reference numeral 101 denotes a control unit that controls the entire server 230, for example, a Central Processing Unit (CPU). Reference numeral 102 denotes a Read Only Memory (ROM) for storing programs and parameters that do not need to be changed. Reference numeral 103 denotes a Random Access Memory (RAM) that temporarily stores programs and data supplied from an external device or the like.

104 is a hard disk or memory card fixedly installed on the server 230, an optical disk such as a floppy disk (FD) or Compact Disk (CD) that can be attached to and detached from the computer device 100, and an external device including a magnetic or optical card or an IC card. It is a storage device. The external storage device 104 can record digital data such as programs and image files. In particular, in the external storage device 104 of the present embodiment, a RAW development processing program described later is recorded as a readable program code in the control unit 101, and the control unit 101 executes the program code.

Reference numeral 105 denotes an interface with an input device such as a pointing device (mouse) or a keyboard 109 for inputting data in response to an operation of a user (server administrator). Reference numeral 106 denotes an interface with the display 110 for displaying the data held by the computer device 100 and the supplied data.

Reference numeral 107 denotes a network interface for connecting to a network line 113 such as the Internet. It is connected to the client 210 through a network line. Further, the server 230 of the present embodiment is connected to the storage 240 through a network line. In the present embodiment, the connection between the server and the storage is a network line, but the connection may be made by any other method. Of course, the RAW moving image file may be recorded in the external storage device 104.

Reference numeral 108 denotes a system bus for communicably connecting the units 101 to 107.

In the present embodiment, an example in which the RAW development processing program is recorded in the external storage device 104 is shown, but the present invention can be applied even when the program is on the ROM 102 or the network.

Next, the operation of the server according to the present embodiment will be described.

FIG. 4 is a flowchart showing a procedure of a RAW development processing program executed by the control unit 101 in FIG.

In step S400, the control unit 101 determines whether or not the RAW development processing program has received any request from the client.

When there is a request from the client, the process proceeds to step S405. If there is no request, the process proceeds to step S485.

In step S405, the process shifts when there is some request from the client.

In step S405, the control unit 101 determines whether the request from the client is a file list request. If it is a file list request, the process is moved to step S410. If it is another request, the process is moved to step S420.

In step S410, the control unit 101 acquires a file list of RAW moving image files recorded in the storage 240.

In step S415, the control unit 101 transmits the file list of the RAW moving image files recorded in the storage 240 to the client 210.

In step S420, the control unit 101 determines whether or not the request from the client is a file name setting. If the file name is set, the process moves to step S425. If it is another request, the process is moved to step S430.

In step S425, the control unit 101 receives the file name of the RAW moving image file to be developed and performs a process of internally holding the file name. When the target RAW moving image file is updated, the development target frame of the RAW moving image is set at the beginning.

In step S430, the control unit 101 determines whether the request from the client is the reproduction information setting. If the playback information is set, the process is moved to step S435. If it is another request, the process is moved to step S445.

In step S435, the control unit 101 receives the reproduction information from the client and performs a process of internally holding the reproduction information.

In step S445, the control unit 101 determines whether the request from the client is the development parameter setting. If the development parameter is set, the process is moved to step S450. If it is another request, the process is moved to step S455.

In step S450, the control unit 101 receives the development parameter information from the client and performs a process of internally holding the development parameter information.

In step S455, the control unit 101 determines whether the request from the client is image data. If it is image data, the process is moved to step S460. If it is another request, the process is moved to step S485.

In step S460, the control unit 101 performs a process of developing image data to be transmitted to the client and a process of determining a transmission method from the internally held reproduction information. Here, the details of step S460 will be described in detail with reference to FIG.

In step S600, the control unit 101 sets the development process to high-quality development. In step S605, the control unit 101 sets the transmission method to image data transmission.

Step S610, step S625, and step S640 are determinations based on the reproduction information.

In step S610, the control unit 101 determines the reproduction state information of the reproduction information. If the playback state information is in the play state, the process proceeds to step S615. Otherwise, the process moves to step S625.

In step S615, the control unit 101 sets the development process to high-speed development. In step S620, the control unit 101 sets the transmission method to streaming transmission.

In step S625, the control unit 101 makes a determination based on the image size information of the reproduction information. If the image size information is less than 50%, the process is moved to step S630. Otherwise, the process moves to step S640.

In step S630, the control unit 101 sets the development process to high-speed development. In step S635, the control unit 101 sets the transmission method to streaming transmission.

In step S640, the control unit 101 makes a determination based on the communication speed information of the reproduction information. If the communication speed information is less than 100 Mbps, the process proceeds to step S645. Otherwise, the process is exited and the process returns to step S465.

In step S645, the control unit 101 sets the development process to high-speed development. In step S650, the control unit 101 sets the transmission method to streaming transmission.

As shown above, the processing of steps S600 to S650 determines the development process and the transmission method. When the video is in a paused state, has a large image size, or has a good communication state, the image data is transmitted to the client in a high-quality state even if the amount of data is large. With this configuration, when it is desired to confirm the RAW development setting with a high-quality display, high-quality display is possible by at least setting the playback state to the pause state. Of course, the methods of development processing and transmission processing may be determined by a judgment other than the present embodiment and a threshold value.

Further, the reproduction information used in the present embodiment is merely an example. For example, information on the image size, bit rate, and compression rate of the transmitted moving image is received from the client side, and the values are equal to or less than the threshold value. Depending on the situation, the configuration may be such that the method of development processing or transmission processing is determined.

Returning to the description of FIG.
In step S465, the control unit 101 develops a development target frame (1 frame) of the RAW moving image according to the development parameters received in step S450. The development processing method at this time is the development processing method determined in step S460. Further, the RAW moving image file to be developed at this time is a RAW moving image file having a RAW moving image file name received in step S425. The development target frame is the first frame in the first development.

In step S467, the control unit 101 updates the frame to be developed. If the playback status information is in the pause state, the development target frame is not changed. If the playback status information is in the play status, the development target frame is advanced to the next frame. However, even in the play state, if the target frame is the final frame, the target frame remains the final frame.

In step S470, the control unit 101 determines whether the method of transmitting the image data to the client is streaming transmission. If the transmission method determined in step S460 is streaming transmission, the process is moved to step S480. When transmitting as one image data instead of streaming transmission, the process is moved to step S475.

In step S475, the control unit 101 transmits the image data obtained by converting the development result into a general image format and the transmission method information to the client. In this embodiment, the development result is transmitted as 12-bit image data. Of course, it may be transmitted in other image formats. For example, it may be transmitted as compressed image data such as JPEG. In addition, step S475 of this embodiment is processing for one frame. When transmitting image data for a plurality of frames, the process of repeating steps S400 to S480 for frames is performed.

In step S480, the control unit 101 transmits the streaming data obtained by converting the development result into a general streaming format and the transmission method information to the client. When streaming in the present embodiment, processing is performed so as to achieve higher compression than when transmitting the image data as it is. In addition, step S480 of this embodiment is processing for one frame. When transmitting streaming for a plurality of frames, the process of repeating steps S400 to S480 for frames is performed. Of course, a method of accumulating data for a plurality of frames, compressing the video, and transmitting the video may also be used.

In step S485, the control unit 101 determines whether to end the RAW development processing program. The trigger for termination is assumed to be a general case such as when the server administrator stops the RAW development processing program (stops providing services to the client) or shuts down the server.

If it ends, the RAW development processing program ends. If it does not end, the process is returned to step S400, and the request process from the client is repeated.

By repeating steps S400 to S485, it is possible to process the request from the client.

With the above configuration, the image corresponding to the set value of the development parameter of the RAW moving image can be displayed and confirmed on the client side.

Further, when it is desired to confirm the RAW development setting with a high-quality display, the RAW development setting can be confirmed regardless of the transmission speed. For example, when a moving image is set to a pause state, high-quality display is possible. Therefore, according to the present invention, when handling a RAW moving image on a server, it is possible to utilize the characteristics of the RAW moving image.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various embodiments within the scope of the gist of the present invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

The present invention is also realized by executing the following processing, that is, providing software (program) that realizes the functions of the above-described embodiment to a system or device via a network or various storage media. , A process in which a computer (or CPU, MPU, etc.) of the system or device reads and executes a program.

Claims (11)

In an image processing system in which a server that develops RAW video and a client that sets parameters and plays back RAW video are connected via a communication network.
The server
Receiving means for receiving development parameters and playback information from the client,
A development processing means that develops RAW moving images according to development parameters, and
The first control means that controls the development process by the reproduction information,
A second control means for switching that controls the transmission method based on playback information,
A transmission means for transmitting the moving image obtained by the development process to the client,
With
The client
Development parameter setting means and
Playback operation means and
Transmission means to send development parameters and playback information to the server,
Receiving means to receive video from the server and
An image processing system characterized by being equipped with. The development process includes a high-speed development process and a high-quality development process.
High-quality development processing is a high-quality and low-speed processing compared to high-speed development processing.
The image processing system according to claim 1, wherein the high-speed development processing is a processing with low image quality and high speed as compared with the high-quality development processing. The image processing system according to claim 1 or 2, wherein the moving image transmission process includes streaming transmission and image data transmission. The playback information is characterized by having at least one piece of information: playback state information indicating the playback state of the moving image on the client, transmitted moving image information indicating the transmission mode of the moving image to the client, and communication speed information indicating the communication speed in the communication network. The image processing system according to any one of claims 1 to 3. The playback state information is information indicating a play state and a pause state, and is
The image processing system according to claim 4, wherein the first control means performs high-speed development processing when the reproduction state information indicates a play state, and high-quality development processing when the reproduction state information indicates a pause state. The image processing system according to claim 4, wherein the transmitted moving image information is information related to the transmitted moving image such as an image size, a bit rate, and a compression rate of the transmitted moving image. The first control means
If either the image size or the bit rate is less than the threshold value, perform high-speed development processing and perform high-speed development processing.
The image processing system according to claim 6, wherein if either the image size or the bit rate is equal to or greater than a threshold value, high-quality development processing is performed. The first control means
If the communication speed is less than the threshold value, perform high-speed development processing and perform high-speed development processing.
The image processing system according to claim 4, wherein high-quality development processing is performed when the communication speed is equal to or higher than a threshold value. The second control means
The image processing system according to claim 4, wherein when the playback state information indicates a play state, the image data is transmitted in a streaming format, and when the playback state information indicates a pause state, image data of one frame is transmitted. The second control means
If either the image size or the bit rate is less than the threshold value, streaming transmission is performed.
The image processing system according to claim 6, wherein if either the image size or the bit rate is equal to or greater than a threshold value, image data is transmitted. The second control means
If the communication speed is less than the threshold value, streaming transmission is performed.
The image processing system according to claim 4, wherein image data is transmitted when the communication speed is equal to or higher than a threshold value.

Images