Why would GE run faster on a dataset with a lot of rows compared to a dataset with fewer rows?

[cosgroveblue]

Do certain expectations tend to take longer than others?

[austiezr]

Hey @cosgroveblue ! Thanks for reaching out. There are a few possibilities.

Whenever possible, GE pushes compute back onto your backend (Spark instance, database, etc.) Instances where that isn't possible, for example when working with local data using the Pandas Execution Engine, may see slower compute when compared to a similar operation taking advantage of greater compute available to your backend.

Additionally, a given expectation may potentially:

• require multiple roundtrips to your datasource
• rely on multiple metrics to be calculated
• include a particularly compute-heavy step

Any of which may cause an expectation to take longer than another which doesn't have those requirements.