Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add function to determine the size of final unixfs DAG given raw bytes #58

Open
masih opened this issue Aug 4, 2023 · 4 comments · May be fixed by #59
Open

Add function to determine the size of final unixfs DAG given raw bytes #58

masih opened this issue Aug 4, 2023 · 4 comments · May be fixed by #59

Comments

@masih
Copy link
Member

masih commented Aug 4, 2023

It would be fantastic to have a function that given the size of raw bytes would cheaply and quickly calculate the size of the final unixfs DAG. This would be useful in cases where the final DAG will be written out into a fixed sized bucket and the user wants to guarantee that each bucket will have the full raw bytes, e.g., FileCoin pieces of size 32GiB.

Other uses of this include pre-allocation of resources or more efficient buffer management depending on the final DAG size for large-scale data intensive applications.

@willscott
Copy link
Collaborator

is what you want the size of the dag when serialized as a car? (including a carv1 header?)

@masih
Copy link
Member Author

masih commented Aug 5, 2023

CAR sections only, for a given LinkPrototype please (though I see that's fixed?, which is fine).

@willscott
Copy link
Collaborator

  • Is there a reason not to include the carv1 header?

  • there are fixed expected cid prefixes for both the raw leaves and intermediate nodes. both types of cids used serialize to 36 bytes.

@masih
Copy link
Member Author

masih commented Aug 5, 2023

Is there a reason not to include the carv1 header?

In my specific usecase, a CAR may contain multiple unixfs DAGs.

willscott added a commit that referenced this issue Aug 6, 2023
…he car file representing a unixfs file of a given size will take

Fix #58
@willscott willscott linked a pull request Aug 6, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants