Fixing the spec with mention about the buffer which cannot be non-con…

…tiguous. (#253)

* Adding specification that the byte buffer needs to be contiguous and
full.

* Fix cargo.lock which messes up benchmarks.

.gitignore

@@ -1,5 +1,6 @@
 safetensors/target
 safetensors/**/Cargo.lock
+bindings/python/Cargo.lock
 *.bin
 *.h5
 *.msgpack

README.md

@@ -94,6 +94,8 @@ Notes:
  They don't really bring a lot of values but are accepted since they are valid tensors
  from traditional tensor libraries perspective (torch, tensorflow, numpy, ..).
  - 0-rank Tensors (tensors with shape `[]`) are allowed, they are merely a scalar.
+ - The byte buffer needs to be entirely indexed, and cannot contain holes. This prevents
+ the creation of polyglot files.
 
 
 ### Yet another format ?

safetensors/README.md

@@ -94,6 +94,8 @@ Notes:
  They don't really bring a lot of values but are accepted since they are valid tensors
  from traditional tensor libraries perspective (torch, tensorflow, numpy, ..).
  - 0-rank Tensors (tensors with shape `[]`) are allowed, they are merely a scalar.
+ - The byte buffer needs to be entirely indexed, and cannot contain holes. This prevents
+ the creation of polyglot files.
 
 
 ### Yet another format ?

safetensors/src/lib.rs

@@ -71,6 +71,8 @@
 //!  They don't really bring a lot of values but are accepted since they are valid tensors
 //!  from traditional tensor libraries perspective (torch, tensorflow, numpy, ..).
 //!  - 0-rank Tensors (tensors with shape `[]`) are allowed, they are merely a scalar.
+//!  - The byte buffer needs to be entirely indexed, and cannot contain holes. This prevents
+//!  the creation of polyglot files.
 //!
 //!
 //!## Yet another format ?