This page is a guide to bringing on-set reference data into ShotGrid. This guide is designed to be a reference and may not apply to every production, but we think it can be a huge help to shows with multiple vendors. While it's simple enough to share all the reference information to a single vendor, when you have multiple it can be more efficient for the vendors when you share only the reference material they need for assigned work.
This process can be very beneficial to a production as it allows:
- Vendors to look up camera, lens, and other on-set metadata information based on Shots they are assigned
- Production or Vendors to generate PDF and CSV Count Sheets and Scan Sheets with the on-set metadata filled out
- Supervisors and production staff to look up on-set material for reference
- Production staff to reverse-find Shots and Plates based on on-set data
- Production to quickly find and send all reference files to a vendor based on material they have been assigned
- Vendors to look up relevant reference files and ask Production to send it to them
This guide is broken into three parts:
- On-set Metadata Import
- On-set Reference File Import
- Linking On-set to Shots
On-set Metadata Import
"On-set Metadata" refers to the descriptive information captured by Data Wranglers, Camera Assistants, Script Supervisors, and others working on set around the shoot. This includes everything from descriptions of what is being shot to technical details about lenses and cameras. This information can be vital to VFX because it allows for more accurate recreation and matching of conditions from the set.
While this seems simple on the surface, there are always issues aligning two databases. For 99% of our productions, they use a variant on the VES Filemaker database, which does an excellent job on set but when transferring to ShotGrid has three inherent issues:
- The FM database combines Slates and Cameras into a single table
- The FM database does not require unique names (as it uses IDs under the hood instead)
- Fields may not match between FM and ShotGrid
This creates a problem when importing as you cannot simply bring the metadata in directly.
There are two methods to importing the metadata, which can help solve the problems above:
Let [RE]DESIGN Do The Work
[RE]DESIGN offers customization options for any production, and that can include syncing metadata from your FileMaker databases to ShotGrid. We can set up a one-time dump from FM to SG or we can set up a weekly process for continuously importing your On-set metadata to SG throughout principal photography and reshoots.
If you wish to have us do the work for you, please create a new ticket with support@artistanywhere.io. Send us a copy of your FM database with some sample metadata in it, and any admin rights so we can look at the underlying structure. We will get back to you with an estimate of the time it will take us to set up the import process. This will be a one-time cost.
Then, when we do the import, we will give you feedback on the quality of the metadata itself. There may be issues with duplicate metadata or misnamed metadata that we spot, and we'll give you a chance to fix before import. The cost of this depends on the quality of the metadata, but generally it goes very quickly for each import.
If you wish to have us do the import, you may skip the rest of this section and go down to On-set Reference Import below.
Export As Tables And Import Into ShotGrid
This is a fantastic solution for when you only need to import once or twice and/or you prefer to have your team do the work. This is also a good choice when you are unclear on the quality of the data and it might need extensive cleanup while importing.
If you wish to do this yourself, please continue reading below!
Self-Importing On-set Metadata
Despite the warnings below, this is generally not a terribly difficult problem and can be done by people new to FileMaker or ShotGrid. Once the process has been sorted out by the production, subsequent imports can go very quickly. Again the caveat here is that the quality of the metadata matters; if the Wrangler did not note every Take or made mistakes this is a great opportunity to catch them before the metadata goes to vendors but it will of course slow you down.
Sample Metadata
[RE]DESIGN has built a sample FileMaker-to-ShotGrid Excel file which you can use as a sample for the below process. Please request our sample file, as it contains instructions for how to do the below process and may be easier to follow when you have sample metadata.
Please create a ticket at support@artistanywhere.io or ask us during our training sessions for a copy of this file.
How to Import FileMaker Metadata into ShotGrid
FileMaker metadata must be exported first before it can be importedm but FileMaker and ShotGrid both can use the same intermediate format to manage metadata, XLSX (Excel file format). In FM, choose File: Export. It will bring up a table similar to the one on the right. Note: this is not a sample export dialog from the VES database.
On the left is a pulldown with a list of tables. You need to determine which Tables to export. Once you choose a Table, move all the fields listed on the left to the right with the Move All button.
Single Table
The FM database often contains a pre-built export template which will be listed in the pulldown.
Note however, this process will create a single export which contains Slate/Camera and Take metadata in one database, as in this sample on the right:
As you can see, because the Takes are really a separate table, FileMaker puts in blank rows in the Slate/Camera table. While it is convenient to have all the metadata in a single table, separating it out later is extra work.
Separate Tables
Alternatively, you will usually see both a "Slates" and a "Takes" table listed in the pulldown section. You can choose those and export them one at a time, which will create two exports, one for each table.
This will save you a step as you will not have to separate Takes from the rest of the metadata later.
Once you have exported out the FileMaker metadata, you can begin to deal with the three problems below.
Self-Import Problem 1: Slate and Camera Combination
As you can see from the chart above, the issue we face is that the "Slate" table in Filemaker must be split into two entities, the Slate entity and the Camera entity.
The FileMaker_to_SG_Import_via_Excel file has instructions which will walk you through how to separate out the Slate/Camera table into two separate SG entities.
Why must we do this?
Below is a sample VES-style FileMaker card, which represents a single record under the hood. It has a layout which very clearly allows the Data Wrangler to import their metadata. However, as you can see, both the Slate (red arrow) and the Camera letter (blue arrow) are together on the same card.
When the Wrangler needs to create a second Camera letter, they duplicate this card. That means that the Slate is in the database twice while the Camera letter is only once.
ShotGrid stores the metadata in a very different manner. Shotgrid stores the Slate separately from the Camera. In the image to the right, you can see the two separate tables: Slates and Camera Setups.
NOTE: If you do have the Takes in a single table with the Slate/Camera metadata, then please follow the instructions in the FileMaker_to_SG_Import_via_Excel file.
Self-Import Problem 2: Unique Names
As you can see from the chart above, the issue here is that FileMaker does not require the user to give every item a unique name, while ShotGrid does.
The solution to this is to create the "missing" metadata before importing into ShotGrid.
Method 1: Filemaker
If you have access to your FileMaker database, it is possible to create a new field which concatenates (combines) two other fields together.
The instructions for concatenation in FM can be found here.
To combine fields in FileMaker:
| Entity | Formula |
| Slates | None needed, use SlateName as provided |
| Cameras | SlateName & "separator" & CameraLetter |
| Takes | SlateName & "_" & CameraLetter & "-" & TakeNo |
In the example above, we assume the fields are "SlateName", "CameraLetter", and "TakeNo", and that the formula would result in 40A_B-1
Once you have created the new columns, be sure to export them with the appropriate tables:
- Camera Names should be exported in the Slates table
- Camera Names (for linking) and Take Names should be exported with the Takes table
Method 2: Excel
In this method we assume that you do not have access to the FileMaker database or cannot make the above change. The FileMaker_to_SG_Import_via_Excel file contains instructions and samples of how to Concatenate, but here are the formulas.
To combine fields in FileMaker:
| Entity | Formula |
| Slates | None needed, use SlateName as provided |
| Cameras | =CONCAT(A2,"_",B2) |
| Takes |
=CONCAT(A2,"_",B2,"-",C2) |
The table above assumes the SlateName is column A, CameraLetter is column B, and TakeNo is column C.
Once you have created the new columns, be sure to include them with the appropriate tables:
- Camera Names should be in the Cameras table
- Camera Names (for linking) and Take Names should be in the Takes table
Self-Import Problem 3: Field Names
It is very likely that you will have fields which do not match between SG and FM, and may need to create additional fields in SG.
Fortunately ShotGrid provides a fairly simple method of managing the field names.
To import into ShotGrid, go to the page with the entity you wish to import. For example, Takes.
Go to the blue "Add Take" button and on the right, choose the drop-down "Import Takes".
This brings up a Bulk Import window where you can copy/paste the cells or drag in the Excel file. We recommend dragging as copy/paste doesn't always get all the records when there is a large number.
After importing, the Step 2: Map Column Names page will appear. Here you will see that it will match columns based on their names. Anything which is not an exact match will come in as "Not importing".
Select the correct field by using the "ShotGrid Column" pulldown under the "Your Column" header. For example, "Take" links to "Take Name".
We recommend using the fields we have provided as much as possible, as they are preset to appear on our reports. Any fields you add will NOT be on our reports and will need to be added both on the ShotGrid shot/plate sheet pages and on our Count Sheet and Scan Sheet PDF exports. You can change the ShotGrid shot/plate sheet pages yourself if you are comfortable with that but you must put in a ticket with support@artistanywhere.io to change the PDF exports.
To add a field, choose "Create Field" from the bottom of the "ShotGrid Column" header.
You can then create the new field. Most likely, you should use fields of type Text, Number (for round integers), Float (for numbers with decimal points), or Date (for dates). Note that you may need to fix the format of dates before importing if they are not detected automatically by SG.
My On-set Metadata Is Imported, Now What?
If you wish to import your On-set Reference files, go to the next section. If you only wish to use the On-set metadata without the Reference files, skip to the Linking On-set to Shots section below.
On-set Reference File Import
Once you have your On-set metadata populated in ShotGrid, you can leverage our Collections via Import Media tool to create links inside SG to your reference files. This allows you to very quickly send your reference files to your vendors when you assign them shots.
These are the items to know about importing reference files:
-
We import the entire folder as a single Version, maintaining all nested folders inside it
- All parent folders must have unique names
Most of the time, shows will import a batch of indivual Slate or Asset folders, with all the child Type folders nested inside. Some shows prefer to import the Type folders instead if they want to have more control over the files after import.
Storing On-set Files
We generally find that most productions will have an on-set drive on set and that is where the Data Wranglers will first put the files. Artist Anywhere will generally provide a backup and sync for the files - but we generally recommend determining when it will be organized.
Pre-Organized
On some shows, the Wrangler will have already cleaned up the files and organized them before copying them to the local on-set drive.
Locally-Organized
However, on many shows the Wrangler is dropping off the raw files and they have not yet been organized. In this case, we recommend the files are copied first to an "unorganized" folder, then the Wrangler or production can copy or move the files into an "organized" folder. In this case, we sync only the Organized folder to the AA Server.
Remotely-Organized
The alternative here is to sync the unorganized on-set data directly to the AA Server, then make a new "organized" folder and copy/move the files into it on remotely via the PCoIP app. This has the advantage of not slowing down your local system and the organization doesn't need to happen immediately if production doesn't have time to get to it right away.
On-set Reference File Organization
For ease of import, you will want your Data Wranglers to organize the data in the most efficent-to-import method possible. However, we often see shows where the reference files have been organized before Artist Anywhere has been contacted, leading to needing to reorganize the data.
The ideal organization for on-set files can vary greatly from show to show, but there are a few things which seem to offer the best potential for import:
- Unique names for every folder
- Organized by hierarchy
- No nested non-unique folders
In the above example, you could import either at the Slate and Asset level or at the Type level as all folders are unique. If you don't wish to import the Type folders, they do not have to be unique.
Problematic imports often have a hierarchy which requires re-organizing before they can be reimported:
- Non-unique folder names
- Organized by type
- Nested non-unique folders
Note that the reorganized folders like LIDAR and SetRef underneath the parent folders 40A and Car are not unique as they are going to be imported as child folders.
Importing Collections
To import, start the Import Media tool from the SG Desktop application.
Drag the Parent Folders into the right side of the app where it says "Drag & Drop Collection".
Your Parent Folders can be the Slate / Asset folders (Car and 40A) or they can be the Type folders (Car_SetRef and 40A_HDRI) as long as they are unique.
If you import the Type folders, on the spreadsheet view of the app, you can link them to the Slate/Asset manually and you can set the "Type" field to match the Type. If you import the Slate/Asset folders, set the "Type" field to something more generic like "On-set".
Note that this tool must process every file in those folders, so it can take a while if you have hundreds or thousands of files. We recommend running this process on your AA Workstation for maximum speed.
Linking On-set to Shots
Once you have your On-set Metadata and On-set Reference Files imported, you will want to enable the automation which allows the metadata and reference files to automatically populate the relevant sections of Shots and Turnovers. This section discusses how that automation happens.
Because of how ShotGrid stores data, we are generally not directly linking Shots to reference photos. While it may look like we are, for instance, linking a Turnover to On-set Reference:
But in reality, we are creating a link that connects all the layers between the Turnover and the On-set Reference through a number of intermediate entities:
More details coming soon!
Comments
0 comments
Please sign in to leave a comment.