Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add special product variables for caching of byte offsets of top-level array elements #21

Open
svniemeijer opened this issue Feb 9, 2017 · 0 comments

Comments

@svniemeijer
Copy link
Contributor

svniemeijer commented Feb 9, 2017

To improve the reading performance of ASCII data and in general of 'array of record' style products (e.g. binary Level 0 data and GOME-2 L1 data) we should find a way to have the byte offsets of top-level array elements cached.

The idea is to do this using product variables. The problem with product variables in its current form is that it 1) requires the length of the variable to be decided before initialization and 2) (requiring a second pass through the product) a full initialization of the variable in one operation to assign all values of the product variable.

We could improve the performance by allowing these two steps to be performed using lazy initialization:

  • allow product variables to start with length 0 and automatically resize the product variable array when higher indexed values are read
  • allowing element-wise initialization (setting the index variable i to the array element that needs to be initialized)
  • use a special 'missing value' (e.g. -1) that will trigger the initialization of that specific product variable array element

Next to this, we then also need to introduce an 'offset expression' for array types (similar to the 'offset expression' for record fields).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant