Malloc error when calling YAML.dump in Ruby 3.2.0

[kmewhort]

I'm seeing an odd process hangs and malloc errors in a call to YAML.dump, only in Ruby 3.2.X.

So far, I'm only able to reproduce it through use of the vcr library -- so I've posted the bug there (vcr/vcr#964) -- but it seems like a deeper issue so hoping someone here might have further insight.

These are the details (reposted from that vcr bug report):

Using:

• Ruby: 3.2.0 (error NOT present with 3.1.2 or earlier; still present on the latest 3.2.1)
• VCR: 6.1.0 (reproducable with earlier); using Webmock, YAML serializer
• OS: macOS Monterey, 12.1

As of upgrading to Ruby 3.2.0, when recording a cassette that gets back a long binary response, the serialization usually completely hangs, and I sometimes gives a malloc error. I can consistently reproduce with the following trivial example:

require "rails_helper"
require "net/http"

RSpec.describe "MallocError", :vcr do
  it "records the cassette" do
    Net::HTTP.get(
      URI("https://uwaterloo.ca/onbase/sites/ca.onbase/files/uploads/files/sampleunsecuredpdf.pdf"),
    )
  end
end

The malloc error that comes up about 1 in 10 runs is:

ruby(25770,0x10532c580) malloc: *** error for object 0x6000095bf5f0: pointer being freed was not allocated
ruby(25770,0x10532c580) malloc: *** set a breakpoint in malloc_error_break to debug

The error occurs at the ::YAML.dump(hash) in lib/vcr/cassette/serializers/yaml.rb:37. You can reproduce by putting a breakpoint there and trying that line.

Interestingly, if at this point in the code, you try serializing just the string but not the full hash, there's a serialization error, but no hang:

str = hash['http_interactions'][0]['response']['body']['string']
::YAML.dump(str)

eval error: expected SCALAR, SEQUENCE-START, MAPPING-START, or ALIAS
  /Users/kent/.rvm/rubies/ruby-3.2.0/lib/ruby/3.2.0/psych/visitors/emitter.rb:32:in `scalar'
  /Users/kent/.rvm/rubies/ruby-3.2.0/lib/ruby/3.2.0/psych/visitors/emitter.rb:32:in `visit_Psych_Nodes_Scalar'
  /Users/kent/.rvm/rubies/ruby-3.2.0/lib/ruby/3.2.0/psych/visitors/visitor.rb:30:in `visit'
  /Users/kent/.rvm/rubies/ruby-3.2.0/lib/ruby/3.2.0/psych/visitors/visitor.rb:6:in `accept'
  /Users/kent/.rvm/rubies/ruby-3.2.0/lib/ruby/3.2.0/psych/visitors/emitter.rb:27:in `block in visit_Psych_Nodes_Document'

If you clobber the string (hash['http_interactions'][0]['response']['body']['string'] = "") it'll pass completely. So the issue arises only in a combination of the invalid, long response string with the rest of the request hash.

Going back to this invalid string...you can get the error with just one character:

YAML.dump(str[10])
eval error: expected SCALAR, SEQUENCE-START, MAPPING-START, or ALIAS

Which is this character:

(ruby) str[10]
"\xC7"
(ruby) str[10].dump
"\"\\xC7\""

So definitely not valid UTF-8, but this the error I would expect:

(ruby) ::YAML.dump("\xC7")
eval error: invalid byte sequence in UTF-8

(ruby) ::YAML.dump(str.dump.undump)
eval error: invalid byte sequence in UTF-8

Two different errors from what looks from the outside be the exact same character. This means there's something funky about that String instantiation itself. I thus suspect the memory in the string has actually been de-allocated or corrupted, especially given the malloc error that comes up sometimes.

I'm not sure if I should raise this issue to ruby itself, but I can only reproduce through VCR...if it's a ruby issue, anyone have insight on how VCR is acting on the String to get into this state (so we can distill it to a simpler repro for ruby)?

[cpgo]

Hi @kmewhort
Have you found any workaround to make this work?
This is possibly the last blocker for our ruby 3.2 upgrade

[kmewhort]

@cpgo, if you're hitting this in VCR cassettes like we were, yes, we found a workaround. The underlying cause of the issue seems to be poor handling of bytes that are non-UTF-8, so you can avoid the serialization issue by using preserve_exact_body_bytes: true in the VCR config (either for the afflicted specs only, or globally).

[cpgo]

Hey! Yeah it was a VCR cassete issue, I should have been more explicit about that.
Thank you, that option is exactly what I was looking for.