Analyzing Millions of rows effectively

#1

Hi -
Basically a product level revenue query has 3M rows. We converted it to a view.

If someone were to pull this information “said view”, then filter by a date range or product or order type, you would be loading millions of rows then filtering by those dimensions.

In most cases I would say that this is not right and it’s a lack of data modeling but conceptual I get it.

One view that is perfect although massive is easier for a user to understand and slice and dice.

So my question is, if I don’t have keys in panoply, and I rely on made up keys in the BI tools, aren’t I still pulling millions of rows in inadvertently to say get to 1 day of data?

Guidance on this would be helpful, to me this is an ELT vs ETL issue.

Thoughts?

#2

There isn’t a query log anymore so im having a challenge diagnosing what the tools are doing in each individual case