Multiple Panoply tables from one S3 bucket


#1

How can I write to multiple Panoply tables from the contents of one S3 bucket? We want the upload to be automated.


#2

That’s a great question.

There are two ways to accomplish that:

  1. Dynamic destination - Our data sources allow you to ingest data into multiple tables if one of the fields in the data holds the “classifier”. For example if we’re talking about events data and you have a field named “event_type” you can split the ingestion into multiple tables based on the type of the events. For that you should input into the destination (under the advanced settings of the data source) the following: events_{event_type}. The “events_” prefix can be changed or omitted based on your need.
    That will ensure that the ingestion will go into the events_* tables where each event type will have its own table. Note that in order for that to work properly all the rows should have values in the specified field.

  2. Multiple data sources - If each file (or set of files) need to be ingested to their own table you can filter the relevant ones (if applicable) using the search or the address path. Each data source will work separately with its own destination in order to ingest the data into separate tables.

In both cases you will need to check the “select all” checkbox in order for the data source to pick up new or modified files in future runs.