Upload Submission

Upload Submission is run by the Vendor to carry out the upload of files from the Vendor to the Production. You should run Upload Submission after running Vendor Submission: Vendor Submission creates the “shipping label” for your Submission, and Upload Submission uses that label to do the actual upload.

Details

Upload Submission searches for pending Submission records created by Vendor Submission, and uses the instructions in those records to carry out the transfer. 

You can manually have Upload Submission search for a single pending Submission, or have it run constantly, automatically transferring any pending Submissions it finds as they appear.  

Upload Submission handles moving files across the internet and uploading them to the Production studio, but it doesn’t ingest the files into ShotGrid or place them in the production filesystem. To complete the Submission process, the Ingest Queue must process the files.

A Note on Version Terminology

The use of the word “version” can be a little confusing in this context. ShotGrid uses “Version” to refer to a piece of reviewable media that is stored and tracked within the ShotGrid webapp and database. But in regular usage, “version” refers to a number denoting a specific revision or iteration of a piece of media. We’ll be using both definitions in this workflow, so to disambiguate, we’ll use “Version” and “version number” in this documentation. 

Submissions and Deliveries

[RE]DESIGN uses the term “Submission” to refer to a transfer of files from Vendor to Production, and “Delivery” to refer to a transfer from Production to Vendor. However, ShotGrid uses an entity called Delivery to track transfers in both directions. We’ll use Submission to refer to the transfer itself, and Delivery when referring to the entity in ShotGrid. For example, “This tool will generate a Delivery record in ShotGrid to track your Submission.” 

Initial Setup

• Install ShotGrid Desktop and log into the appropriate ShotGrid site
• Use Vendor Submission to create a shipping record for the files you wish to transfer. 

User Interface and Workflow

Launch Upload Submission

Open ShotGrid Desktop, select your project, and click on Upload Submission.

Initial View

When you first open the app, the submission list will be empty and you’ll be prompted to check for pending Submission records.

Press the Start Search button to check for new pending Submissions every 30 seconds or the Check Now button to do a single one-time check.

List View

Vendor Submission creates Delivery records in ShotGrid with the Status of Ready to Upload.

As soon as you hit Check Now or Start Search, Upload Submission will begin looking for these records.

As soon as the app finds a pending Submission record, it will display it in the list view and immediately begin uploading it. 

The Upload Submission app will only take one Delivery from ShotGrid at a time, starting with the oldest. If you wish to order your uploads, create the Deliveries in the order you wish to upload. When you click Start Search, a second Delivery in ShotGrid will be picked up by Upload Submission when the first one is completed.

Note that the user who launches the Upload Submission app and the user who runs the Vendor Submission app must belong to the same Vendor Group.

To stop an upload from running, simply close the Upload Submission app.

 

To restart the upload, open the app and click Start Search again. Depending on the transfer method, the upload may be able to pick up where it left off.

If you are using the date of upload to determine the upload path, you’ll need to start the upload over if the day changes while uploading is paused.

Production Workflow

When a Submission has finished uploading, Upload Submission will set both the Status and the Delivery Progress fields of its Delivery record in ShotGrid to Delivered. On the Production side, the Ingest Queue(link) will see the Delivered Status and process the Submission.

Status and Delivery Progress

As a Submission moves through the transfer process from Vendor to Production, the Status and Delivery Progress fields on its Delivery record change several times to reflect the current stage in the process. Here are the different states of the two fields.

The values in the table below are applicable to Cyclone transfers only. If you are using a different transfer method, the values may vary.

Status	Delivery Progress	Description
Ready to Upload (rupd)	Ready to Upload	Vendor Submission is first run; Delivery created
In Progress (ip)	Uploading	Upload Submission starts uploading the Delivery
In Progress (ip)	In Vendor Bucket	Uploading has completed
In Progress (ip)	In Transit to Central	Delivery is getting copied via AWS Batch
In Progress (ip)	Queued to Download	Delivery is in queue to download to production storage
In Progress (ip)	Downloading	Delivery is downloading to production storage
In Progress (ip)	In Central Storage	Downloading is complete; Delivery is available in production storage
Delivered (dlvr)	Delivered	Delivery process is complete; Delivery is ready for ingest

 

Running Upload Submission on Multiple Computers

You can run Upload Submission on any machine, provided that the input files are accessible from that machine.

Upload Submission looks for all Delivery records with a Status of Ready to Upload. If two users are creating Submissions simultaneously on different machines, and the files of one user’s Submissions are not accessible from the other user’s machine, this may result in an error.

Troubleshooting

Error Logs

Sometimes [RE]DESIGN will ask you to view or send us the ShotGrid Desktop app's error logs. To access them, open ShotGrid Desktop. In the pulldown menu in the upper right corner, select "Open Log Folder". Please ZIP up the all the files in that folder and send them to us with your ticket, or email them to [e-mail].

Connection Issues

You may need to update your firewall rules if you have having trouble connecting to SG or uploading files. Please pass the following along to your IT department:

• ShotGrid: General Security Ecosystem Documentation 
• If you are using Cyclone, there will be an AWS Bucket assigned solely to your facility. The bucket name will generally match the pattern of productionname-vendorname,which you can reach at s3.amazonaws.com/bucketname. If you’re unable to access your AWS Bucket, please contact us at [e-mail].
• If you are using Aspera on Cloud, you will need to open up studioname.ibmaspera.com. You can get more information from IBM’s Aspera on Cloud documentation.
• If using other services including Aspera Enterprise, NightRaven, or Content Hub, please contact us at [e-mail]. 

Upload Issues

Uploads can sometimes fail, often due to either connection issues or the uploading computer lacking the needed specifications to run the tools. In such cases, you’ll see a Status of Error and a Delivery Progress of Upload Failed on the Delivery record in ShotGrid. 

To resubmit a failed upload, follow these steps:

1. Go to the Deliveries page in ShotGrid and reset the Status and Delivery Progress fields of your Submission’s Delivery record back to Ready to Upload. 
2. Launch Upload Submission, and click the Check Now or Start Search button. 

Please do NOT create a new Submission, as this will require re-uploading all existing files. In addition, if the new Submission’s Delivery record has the same name as the previous one, the files will conflate and the production will get an error when ingesting.

Here are some known upload errors:

• 400 and 500 errors: If you receive a numbered error such as 404 or 502, that means that the upload tool lost contact with the server. These error codes are often accompanied by messages like HTTP Error, Client Error, or Protocol Error. Please reset the Status and Delivery Progress fields of your Submission’s Delivery record in ShotGrid back to Ready to Upload and try again.
• CRUD error: This is a ShotGrid-related error. It means that the user does not have the necessary permissions to create the entities in SG. Please reach out to the Production to confirm that you are properly configured as a user.
• Failed to copy / not found errors: If in ShotGrid or in the log file you see the message "Failed to copy: directory not found" or "Transfer failed: No such file or directory", this means the source files were moved between the creation of the Submission and the upload. Restore the files to their original path and set the Status and Delivery Progress fields of your Submission’s Delivery record back to Ready to Upload and try again.
• File size mismatch: Vendor Submission stores the size of each file in the Submission, and as a validation step, Upload Submission compares it to the file size at upload time. There could be a file size mismatch, for example, if files need to be copied to an IO machine before transfer to Production. If Upload Submission runs before the copy has completed, the partially copied files will cause a file size mismatch. To eliminate this error, Once the files have completed copying, create a new Submission with a different name.
• App is crashing: If the Upload Submission app, Python, or ShotGrid Desktop itself is crashing consistently, then the computer may be too old or underperforming. We have found that 16GB of RAM and a newer computer (within the last 5 years) and operating system (within the last 5 years) are often required to ensure stability. Sometimes you'll receive "Out of memory" or "Cannot allocate memory" errors as well when your machine meets specs but if there are other apps running on your machine. If you continue having issues, please reach out to us at [e-mail] and to your production.