Ingest Queue

The Ingest Queue completes the work of bringing media files into ShotGrid and the production filesystem, whether they originated from Vendor Submissions or internally.

Details

The Ingest Queue is the final step in getting media files into ShotGrid and the production filesystem, whether they were uploaded by a Vendor via the Vendor Submission and Upload Submission apps, or originated internally and submitted via the Internal Submission app.

Typically the Ingest Queue runs continuously in the background, looking for Submissions that are ready to ingest and processing them automatically as they come in. You can also direct the Ingest Queue to manually to process a single Submission.

When the Ingest Queue receives a Submission, it validates the files within it, copies them into the production file system, and creates records and web-playable versions of each in ShotGrid.

The Ingest Queue only works with Submissions that originate from Vendor Submission or Internal Submission. To create records in ShotGrid for media without creating a Submission, you can use Import Media.

Initial Setup

Install ShotGrid Desktop and log into the appropriate ShotGrid site
Create a Submission, either through Vendor Submission and Upload Submission, or Internal Submission
To receive notifications by email, ensure that you have Delivery email notifications turned on in your ShotGrid account settings

To turn on email notifications in ShotGrid for Deliveries, go to the Account Settings -> Email Notifications tab and then check Email me about Deliveries that I'm involved in (those that I've created or replied to, or where I'm a recipient, except for file attachments).

You can learn more in the Email Notifications section of the ShotGrid User Guide.

User Interface and Workflow

Launch the App

Open ShotGrid Desktop, select your Project, and click on Ingest Queue.

Initial View

Once you’ve launched the app, you’ll see this screen with two buttons at the top, “Start” and “Check now”, and an area for manual drag and drop of Submissions.

There are several ways to provide Submissions to the Ingest Queue for processing:

Continuous auto-discovery of Submissions (preferred)
One-time check for Submissions
Manual input of Submission

Continuous Auto-discovery of Submissions

The most common use case for the Ingest Queue is to have it running continuously in the background. To start the app in this mode, simply click the Start button. It will check for new Submissions every 30 seconds. It will begin processing any Submissions it finds, up to a configurable maximum quantity, and queue up the rest.

There is a maximum quantity of simultaneous ingests to prevent overloading ingest machines and/or ShotGrid. By default, the Ingest Queue will process a maximum of two Submissions at one time, but this value is configurable. Contact us at support@artistanywhere.io to change it.

To pause the Ingest Queue from scanning for New Submissions, click the Pause button. To stop it, simply close the app.

One-time check for Submissions

You can also have the Ingest Queue query ShotGrid for new Submissions only one time without continuously running. To do so, click the Check Now button.

Manual Input of Submission

If you want to bypass the app’s scanning capability and manually submit a Submission for ingest, you have a couple options:

Within ShotGrid, right-click on a Delivery record and choose Ingest Queue. This will launch the app with the selected Submission pre-loaded.

As a less common workflow, you can drag or drop either a folder containing a Submission or the manifest.yml file from a Submission onto the drop area of the app.

Transfer Settings Dialog

If you are using a transfer method that requires adding credentials, such as Aspera on Cloud Files or Aspera Faspex, you will receive a popup the first time the app receives a Submission. Follow the instructions for your transfer method from the Vendor Submission Transfer Methods document.

To bring up the Transfer Setting Dialog at another time, click the gear button to the right of Start and Check Now.

The Ingest Process

Once the app finds or receives new Submissions, it will immediately begin the ingest process.

To return to the drag and drop area from the Ingest Status view, click the +Delivery button.

Once the Ingest Queue receives a Submission, it does the following:

Downloads media from a remote server, if necessary
Validates media to ensure that it’s complete and undamaged
Copies files into the production file system, placing them within a pre-configured folder structure and creating any necessary folders

As you ingest files via the Artist Anywhere Studio Workflow Tools, the Ingest Queue app copies them into your production filesystem, creating folders as needed to gradually build it out. The tools have a default folder structure, but you can customize it to suit your studio’s needs. To make modifications to your folder structure configuration, contact support@artistanywhere.io.

Within ShotGrid, it:
- Creates PublishedFiles, per-file records for tracking files on disk
- Creates Versions, web-playable media and thumbnails
- Creates a per-Delivery Playlist for review
- Imports any metadata, such as the Submission Note, accompanying the Submission
- Updates Delivery Status

Note that while the From, To, and Created By fields on the Delivery in ShotGrid will be set when Vendor Submission or Internal Submission is run, the PublishedFile, Version, and Playlist records that are created by the Ingest Queue will have their Created By fields set to the currently logged in user.

If there are already files with the same name in the destination folders, the app will ask you if you want to overwrite them.

Monitoring Ingest Progress in ShotGrid

Typically, the Ingest Queue will be running in the background on an ingest server, but you can monitor the status of a Submission from any machine by looking at its Delivery record in ShotGrid.

For more information about Delivery records, see Autodesk’s documentation.

Status and Delivery Progress fields: Throughout the transfer and ingest process, the Artist Anywhere Studio Workflow Tools will update the Status and Delivery Progress fields on the Submission’s Delivery record. You can look at these fields in a list page for a quick status update.
- When a Submission has been delivered and is ready for ingest, its Status and Delivery Progress will both be Delivered.
- When ingest is complete, its Status and Delivery Progress both change to Received.
- If there was an error in the ingest process, the Status will be Errored and the Delivery Progress will be Ingest Failed.

Reply thread: For more detailed progress information, look at the details page for your Submission’s Delivery record. Ingest Queue and all of the Studio Workflow Tools add replies and events to the thread at every stage of the process, with status updates and links to attached files like manifest.yml and log files.

Log files: The Ingest Queue generates the log files ingest.log and transcoding.log, which contain verbose logging of the app’s processing. You can access these from the reply thread on the details page for your Submission’s Delivery record.

Ingest Completion

The app will indicate when ingestion has completed for a Submission. You can click on the arrow on the right of each Submission to see details about the files within it. ShotGrid will also send email notifications to users in Groups mentioned in From, To, and Cc for the Delivery.

ShotGrid’s email notifications when a Submission gets ingested contain the Delivery name and Project name in the subject, and Delivery details in the body of the email.

Additionally, when a Submission has completed ingestion, the Status and Delivery Progress fields on its Delivery record in ShotGrid will change from Delivered to Received.

Troubleshooting

The suggestions below will help you fix some issues that come up when using the Ingest Queue. If you come across any issues that you can’t diagnose and fix yourself, please contact support@artistanywhere.io.

Error Logs

Sometimes [RE]DESIGN will ask you to view or send us the ShotGrid Desktop app's error logs. To access them, open ShotGrid Desktop. In the pulldown menu in the upper right corner, select "Open Log Folder". Please ZIP up the all the files in that folder and send them to us with your ticket or email them to support@artistanywhere.io.

Firewall Configuration

If you are running the Studio Workflow Suite on your own network, you may need to update your firewall rules if you have having trouble connecting to ShotGrid or uploading files. Please pass the following along to your IT department:

ShotGrid: General Security Ecosystem Documentation
If you are using Aspera on Cloud, you will need to open up studioname.ibmaspera.com. You can get more information from IBM’s Aspera on Cloud documentation.
If using other services including Aspera Enterprise, NightRaven, Content Hub, Signiant, Media Shuttle, or SFTP, please contact us at support@artistanywhere.io.

Connection Errors

You may experience issues with connecting to your ShotGrid website, AWS, Aspera, or any studio-specific file transfer service. If you do, they should display an error in the Ingest Queue app, on the delivery record in ShotGrid, or in the error log.

URL, 10X, 40X, and 50X errors: An error with a numbered error code such as 104, 404 or 502 means that the Ingest Queue lost contact with the server. This is often accompanied by messages like "urlopen error", "HTTP error", "Client Error", or "Protocol Error". To fix this, on the Submission’s Delivery record in Shotgrid, set the Status and Delivery Progress fields back to Delivered and the Ingest Queue will try again.
CRUD error: This is a ShotGrid-related error. It means that the user does not have the necessary permissions to create the needed entities in ShotGrid. Please confirm that the user who is running ShotGrid Desktop has permissions to see all fields and entites.

If you are using Aspera, resetting the Submission’s Status and Delivery Progress to Delivered will requeue the Submission for ingestion. However, the Submission will be downloaded again at ingest time. To prevent re-downloading, if you are certain that the local files are good, simply drag the Submission folder back onto the Ingest Queue app to rerun its ingestion.

Local Path Errors

You may come across errors related to expected paths in the file system to which you’re copying ingested files. For external (Vendor) Submissions, the source folder is generally /deliveries (or /downloads, /postoffice, etc.); for Internal Submissions, it can be any path the user specifies. Below are a few we have seen. The fix for most of them is to address the path issue; if you can fix the path for the Ingest Machine, set the Status and Delivery Progress fields on the Delivery for the Submission back to Delivered; if not, a new Submission may be necessary. If you have any questions, please reach out to support@artistanywhere.io.

Permission Denied: The user does not have access to the folder containing the files to be ingested
Source file does not exist on disk: The Ingest Machine can’t see the source files. If the source files were on the computer that ran the Submission app, it's possible the Ingest Queue cannot see the folder they are in. Similarly, when using services like Box or DropBox, the path often includes the username, so even if Box is mounted on both machines, the path might be different. If this is the case, you should copy the files to your server before ingesting.
The system cannot find the path specified: Same as above.
The network path was not found: Same as above.
Bad file descriptor: This can affect Artist Anywhere users who have dehydrated the source files. Please ensure all files are hydrated first.
No space left on device: Either the server or the temp drive (i.e., C:\Windows\Temp on Windows) on the Ingest Machine has run out of space.

Template Errors

Our tools use templates to determine file naming and folder structure. Occasionally template errors arise, such as these:

Can't evaluate template ____, keys [ ____ ] are missing: The ShotGrid entity linked to the media file being ingested is missing a value required to create a folder. For example, to create folders for a Shot, the Shot must have a Sequence assigned to it in ShotGrid, or to create folders for an Asset, the Asset must have an Asset Type assigned.
Not a File Sequence, List index out of range, or frame_range error: These errors are typically seen when a file is sent with extra "." in the name. Our tools recognize the "." before the extension, and can only recognize one more "." when a frame range is included. One of these errors would be generated if a file included an extra "." but did not get recognized as a file sequence (e.g., myfile.####.exr).
Couldn't find a template for entity type _____ and format _____: This means someone sent you a file which is accepted but not expected for the particular Entity. For instance, you may configure receiving a .cube file for a Shot but it was not configured for an Asset. Reach out to the submitter to have them resubmit if this is not desired, or to us at support@redesign-group.com if the new template is required.
Format _____ is not contained inside format group _____: Similar to above, please reach out to use to configure.

Manifest Errors

Our tools include a file called manifest.yml, which carries all of the technical details about the Submssion. This is a required file and errors can sometimes arise when the user does something unexpected with this file. Reach out to us at support@artistanywhere.io if you encounter any of the following and cannot address it yourselves:

Codec can't decode byte: When a Vendor drops in a CSV that contains non-Roman characters in our older tools, this error occurs. You can fix it by either creating a new Submission from the source files or having the Vendor resubmit. As of December 2021, all languages should be accepted; if you still receive this error please reach out to us at support@artistanywhere.io.
Delivery mismatch! The given Delivery was intended for a Delivery with id ______, current one's is ______: This occurs when a user creates a second Delivery with the exact same name as an earlier one, and the manifest.yml file had been uploaded already. Deliveries are not overwritten when uploading, so the manifest.yml will be from the earlier Delivery. Please ensure that your users are using unique names for all Submissions, then reset the Status and Delivery progress fields on the Delivery record to Ready to Upload, and use Upload Submission to reupload the the Submission.
Size is wrong for ______, got ______, expected ______: This happens when the user tries to ingest a Submission that is still in the process of uploading or the file is swapped before uploading. The Submission apps store file size in manifest.yml; if the size changes at any point after the Submission apps are run, the user will receive an error when ingesting. To fix this, create a new Submission with Vendor Submission or Internal Submission.

Transcoding Errors

Occasional errors occur during the transcoding process, and they fall into two general categories:

Upload issues: Sometimes ShotGrid will error out when trying to upload a transcoded file. This generally happens on a single item like a thumbnail or filmstrip. If the final Versions seem fine, you can change the Status and Delivery Progress fields on the Delivery record to Received and consider it good. If you wish to run it again, set the Status and Delivery Progress back to Delivered.
Transcoding issues: There are files which our transcoder cannot handle, and it will fail when attempting to transcode them. Sometimes it is a bad file, sometimes the image is too large, and sometimes a movie is a codec which our tools cannot understand. Feel free to reach out to us at support@artistanywhere.io in this case. One possible workaround is to manually add the source file to the Uploaded Movie field in ShotGrid after Version creation, in case ShotGrid’s built-in transcoder has different results. If that is done, set the Status and Delivery Progress fields on the Delivery record to Received.

Common Aspera Errors

Some errors are unique to Aspera on Cloud (hosted by IBM) and Aspera Enterprise (hosted by the Studio). Feel free to reach out to us with any of the below at support@artistanywhere.io. Here are a few common ones:

No such file or directory: This happens when the expected path is not found on the Aspera server. Often this is because the Vendor Remote Folder or Aspera Remote Folder on the Group record is not configured correctly in ShotGrid. If that is the case, correcting the folder name on the Group will allow the Delivery record’s Status and Delivery Progress fields to be set back to Delivered, and therefore reingested by the Ingest Queue.
Insufficient permissions: This arises when a user does not have permissions to see the folder that stores the files they are uploading or downloading. This generally requires talking to the Studio's Aspera support team to correct.
Connection aborted: This is usually a timeout response from Aspera. It is possible that it is a short-term problem, in which case setting the Delivery’s Status and Delivery Progress fields back to Delivered will address it. If you get the same error a second time, please reach out to both us and your Aspera support team.
Cannot allocate memory: This is an issue with the Aspera ASCP application, which our tools leverage under the hood. This type of error generally means that the computer itself has run out of RAM; close all other applications or use a computer with more RAM.
Session token ____ has expired: This generally means something has changed with the user on Aspera and they cannot access the folders any more. Reach out to your Aspera support team about this.
Float division by zero, Socket is not connected, Device not configured: These errors seem to point at a permissions issue with Aspera command-line writing to disk on MacOS. In the Mac's System Preferences > Security & Privacy settings > Privacy, go to Full Disk Access, then allow Terminal