SDK data sources - how long till data appears?



How long should I expect for it to take for data to appear in tables when using the Panoply SDK data sources?


How long it takes for data to appear in tables depends on a number of factors, including whether it is the first time you are setting up the data source, what kind of data source you are working with, and whether you are using the Panoply UI or an SDK.

When you add a data source through the Panoply UI, the first load can be a bit slow if it involves a historical load with a lot of data. Subsequent loads from the same data source should be faster because we only collect data that has been added or updated since the last load.

The kind of data source you’re working with also affect load times. APIs are subject to external rate-limiting, which makes it difficult to calculate how many minutes it will take to load data into your tables.

When using the SDK, if the data accumulates at a slow speed (because the source is slow, for example), there’s a timer that will flush it every 20 minutes. This delay also applies to SDK ingestion when it is configured correctly. The number of other jobs being run in the system can also affect data ingestion speed.

The SDK uses an internal buffer to store events and actually sends them once the buffer is full or a timeout is reached. Once the data is actually sent it can take up to 20 minutes. That really depends on the amount of data that you sent. The more data you send the faster it will appear in the tables.