Technical reference

QI Concepts and User Manual

This chapter assumes that a production has been created and that users and owners have been assigned to the production.

Log in

  1. Click on “Login” button and follow the instructions on the screen.
  2. In the drop-down boxes, select organization and production. Only accessible organizations and productions are shown.
  3. Click on “Use” button.
  4. The main window opens.

Note: Actions under the “CREATE NEW” heading are accessible only to administrators. These will fail when run by “normal” users.

Main window

Initial project configuration

Before ingesting data, the following must be configured.

  1. Global settings, menu “Project →Global Settings”. The most important options to set are default ingest and output paths; these are explained in detail in section “Path Template” on page 2.
  2. Primary destination must be added (see section “Definitions” on page 1), menu “Project → Destination editor”. Additional “backup” destinations may be added as well.
  3. An ingest workflow must be instantiated, menu “Project → Workflow editor”.

Important:the workflow must be explicitly saved by choosing “Actions → Save” in the editor dialog. Clicking the “Close” button only closes the dialog without making any changes.

Ingesting data

Drag a file or folder to the “drop area” in the main window and follow the instructions on screen.

Important points:

  • Only a single item (file or folder) may be dropped onto the area. Trying to drop multiple items will be rejected.
  • On Windows systems, trying to ingest from a disk root (corresponding to path such as D:\) will result in an error. This will be fixed in a future release.

Concepts

This chapter defines the main terms used throughout QuineIngest and QuineCore.

Definitions

DestinationA folder on any disk that was added as a volume to a specific project using the QI Volume Editor.
DeviceA user defined ‘thing’ used to group files. A device is defined by a device class
(Camera, application, audio recorder, …),
The device Manufacturer and Model (for example ARRI, Alexa Mini).
Purpose 
LogicalResourceThe common representation of multiple physicalResources that all
originate from the same physicalResource. For example an original
video clip, its edit proxy and daily transcodes will all share the same LogicalResource.
PhysicalResourceThe representation of a specific resource. Each PhysicalResource
will have it’s own purpose. If you have a video clip, this clip may be
transcoded for different purposes such as an edit proxy and a daily.
Each of these transcodes will be represented by their own
PhysicalResource where that purpose is defined, but all these different
PhysicalResources will have one common LogicalResource where data
that is common to all the PhysicalResources, such as TC or clip name, is stored.

Path Template

A path template is a path to where files will be placed. The path can combine user specified components with predefined template values which during ingest will be replaced by relevant values for the data being ingested.

When building the template path you separate folders with a forward slash ‘/’.

Template arguments are defined as $(template)

When configuring device paths in the Ingest Path Editor, the valid template values are:

$(CatalogDate)will be replaced by the date the user specifies when starting an ingest job. Format: yyyy-mm-dd 
$(DeviceType)Will be replaced by the type of the device selected for the ingest job
$(DeviceManufacturer)Will be replaced by the manufacturer name of the device selected for the ingest job
$(DeviceModel)Will be replaced by the model name of the device selected for the ingest job

When configuring a node that can output files in the Workflow Editor, the valid template values are:

$(LogicalResourceDate)Will be replaced by the yyyy-mm-dd of the LogicalResource. This will be the date
the user specified when ingesting the data, typically the recording date of the
data, but if a user at a later point changes the data of one specific clip, the
LogicalResource date may no longer be the same as the CatalogDate
$(PhysicalResourceDate)Will be replaced by the yyyy-mm-dd of the output physicalResource.
This will be the date of the creation of this physicalResource, which
is not necessarily the same date as when the video clip was originally recorded
$(Purpose)Will be replaced by the purpose of the output resource
$(DeviceType)Will be replaced by the type of the device selected for the ingest job
$(DeviceManufacturer)Will be replaced by the manufacturer name of the device selected for the ingest job
$(DeviceModel)Will be replaced by the model name of the device selected for the ingest job

A template path for a device may, as an example, be defined as: 

MEDIA/$(DeviceManufacturer)/$(CatalogDate)

If you then ingest some RED data and that data was recorded 2019-01-01, the complete path will become:

MEDIA/RED/2019-01-01

$(DeviceManufacturer)was replaced with RED, and $(CatalogDate)was replaced with 2019-01-01

Another template path for an FFmpeg node configured to ouput edit proxies, with a user defined purpose of EDIT_PROXIES may be defined as:

MEDIA/$(Purpose)/$(LogicalResourceDate)

 

Destination Editor

The volume editor is used to create volumes for a production. 
When adding a new volume you browse to a folder, specify if the volume is a primary or backup volume (irrelevant at this stage), and you give your volume a name. The name should be descriptive in such a way that you can recognize the name when setting up workflows later.


After adding a volume, a file named QI-VOLUME-ID.TXT will be created in the selected folder. This file must never be removed or altered. 

Volumes can only be removed if no data has been ingested to that specific volume. As soon as any data is ingested on the volume, it is currently not possible to remove it from QI. The folder can technically be deleted manually, but that will very likely cause downstream issues when using QI or QuineCore later.

Ingest Path Editor

In the Ingest path editor you configure where files from a given device should be placed. When adding a new device you specify its class, meaning what “type” of the device it is, and the Manufacturer and model name. You then specify where on a Volume files with this device should be placed. The location is defined as a Path Template.

You do not need to pre-configure all devices in this dialogue. You will be able to configure a device during the ingest startup process.  

Download Assets

QI can be used to continually download files matching a set of given purposes and a date range by opening File->Download Assets.

Under the DownloadSource node you specify the date range from First date to Last Date, and select one or several purposes that you wish to download.

You must then configure which volume the files should be downloaded to in the BlobDownload node. The Path Template in this node is a fallback template path. This means it is only used if a file that is to be downloaded does not have any locations tracked in the database that can be copied. In all cases where the file already exist on a volume somewhere, and is tracked by QuineCore, the same path will be used for the downloaded file on the download destination. Files that does not have such a path that can be copied are typically AMS output files, as they are generated and stored in the cloud.

RedLine Configuration

Workflow Editor

When opening the workflow editor you can do the following

Create NewCreates a new Graph from the list of provided template graphs
Edit ExistingOpens an existing graph so that it can be edited and saved
SaveSaves the graph that is currently being edited
DeleteDeletes the graph that is currently being edited
Import TemplateAllows you to browse to an exported template graph on disk
Export TemplateAllows you to export a graph as a new template and save it to disk.

When you have opened a graph, either with Create New or Edit Existing, there will be several nodes on the left side of the window. When selecting a node, the configuration for that node opens on the right side. 

Template Workflows

We have 2 workflow templates

  • [NewDataTemplate]
  • [ExistingDataTemplate]

[NewDataTemplate] graphs should be used for any new data that does not yet exist in a volume for the project, meaning you want the data to be copied and placed within the project volume in a structured manner.

A [NewDataTemplate] graph should never be used to ingest or process data that already exist in any volume associated with the project.

[ExistingDataTemplate] can only be used for data that already exist inside a project volume destination. It can be used on data that was ingested with a [NewDataTemplate] graph, as well as on data that was manually placed on the volume.

[NewDataTemplate]

When creating a new graph using [NewDataTemplate], you can configure the following nodes as described:

  • LocalCopy
    • Select the Volume(s) that ingested data should be copied to. 
  • MovieSlate
    • Swap scene/slate. This is only relevant if ingesting MovieSlate files, and if the option is checked, the scene/slate values will be swapped when stored in QuineCore.
  • Ffmpeg
    • Transcode original video: Enable/Disable transcoding of input data
    • Output definition
      • Volume specifies to which volume transcoded outputs should be stored
      • Path Template lets you define where on the specified volume transcoded outputs will be stored
      • Purpose is the user-defined purpose of the transcoded outputs. You can filter by purposes on QuineCore later to find files matching your specified purposes (not yet added to QuineCore).
    • The rest of the options are fairly standard transcode parameter settings.
  • BlobUpload

(Original refers to the data that was given to QI, and does not include data generatedby QI, such as Ffmpeg output)

  • Original Video: When selected, all video data ingested with this graph is uploaded.
  • Original Audio: When selected, all audiodata ingested with this graph is uploaded. 
  • Transcode output: When selected, anything transcoded by QI will be uploaded.
  • All other files: When checked, Files that are not transcode output, and not recognized as original video or audio, will also be uploaded.

Checking all other files option alone is not enough if you want to upload original video/audio or transcode output as well. All other files only denotes other files. You can none, one or several of the options at the same time. A graph to upload all original data would select Original Video, Original Audio and All other files, while a graph to upload original sidecar files, but none of the media, as well as the transcoded files, would select Transcode output and All other files.

  • StreamJob (can only select one)
    • None: No streams are generated
    • Original Input: streams are generated from original video, if they are uploaded
      Transcoded output: streams are generated from transcoded files, if anything is transcoded.
  • Receipt
    • Specify the volume and location on the selected volume to where receipts for completed ingests should be written.

[ExistingDataTemplate]

This template has all of the same nodes as [NewDataTemplate] except that instead of a LocalCopy node it has  ResourceTracker node

  • ResourceTracker (temporarily named FindExistingResources)
    • UntrackedFilesPurpose: All files that were not already tracked by QuineCore will receive this Purpose.