Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop rescuing all exceptions. #2063

Merged
merged 2 commits into from
Oct 7, 2015
Merged

Conversation

myronmarston
Copy link
Member

We do want to generally rescue Exception and
any subclasses but there are a few exceptions
we should not rescue. For example, if we rescue
SystemExit, it prevents exit from exiting.

Attempted fix for #2058.

Note that the spec I added failed without these changes and passes with it...but there are some other weird effects that I do not understand:

➜  rspec-core git:(stop-rescuing-all-exceptions) bin/rspec spec/integration
Run options:
  include {:focus=>true}
  exclude {:ruby=>#<Proc:./spec/spec_helper.rb:106>}

All examples were filtered out; ignoring {:focus=>true}

Randomized with seed 9279
...........

Finished in 2.12 seconds (files took 0.11962 seconds to load)
11 examples, 0 failures

Randomized with seed 9279

.......

Finished in 2.3 seconds (files took 0.11962 seconds to load)
18 examples, 0 failures

Randomized with seed 9279
➜  rspec-core git:(stop-rescuing-all-exceptions) bin/rspec --seed 33805
Run options:
  include {:focus=>true}
  exclude {:ruby=>#<Proc:./spec/spec_helper.rb:106>}

All examples were filtered out; ignoring {:focus=>true}

Randomized with seed 33805
..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................*.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) RSpec::Core::Metadata for an example points :example_group to the same hash object as other examples in the same group
     # Cannot maintain this and provide full `:example_group` backwards compatibility (see GH #1490):(
     Failure/Error: expect(b[:description]).to eq("new description")

       expected: "new description"
            got: "group"

       (compared using ==)
     # ./spec/rspec/core/metadata_spec.rb:138:in `block (3 levels) in <module:Core>'
     # ./spec/support/sandboxing.rb:14:in `block (3 levels) in <top (required)>'
     # ./spec/support/sandboxing.rb:7:in `block (2 levels) in <top (required)>'

Finished in 8.28 seconds (files took 0.7278 seconds to load)
1399 examples, 0 failures, 1 pending

Randomized with seed 33805

....................................................................................................................................................................................................................................................................................................................................................................................................................

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) RSpec::Core::Metadata for an example points :example_group to the same hash object as other examples in the same group
     # Cannot maintain this and provide full `:example_group` backwards compatibility (see GH #1490):(
     Failure/Error: expect(b[:description]).to eq("new description")

       expected: "new description"
            got: "group"

       (compared using ==)
     # ./spec/rspec/core/metadata_spec.rb:138:in `block (3 levels) in <module:Core>'
     # ./spec/support/sandboxing.rb:14:in `block (3 levels) in <top (required)>'
     # ./spec/support/sandboxing.rb:7:in `block (2 levels) in <top (required)>'

Failures:

  1) When an example forks and exits it the subprocess prints the results only once
     Failure/Error: expect(finished_lines.count).to eq(1)

       expected: 1
            got: 2

       (compared using ==)
     # ./spec/integration/subprocess_exit_spec.rb:21:in `block (3 levels) in <top (required)>'
     # ./bundle/ruby/2.1.0/gems/aruba-0.6.2/lib/aruba/api.rb:46:in `chdir'
     # ./bundle/ruby/2.1.0/gems/aruba-0.6.2/lib/aruba/api.rb:46:in `in_current_dir'
     # ./spec/integration/subprocess_exit_spec.rb:19:in `block (2 levels) in <top (required)>'
     # ./spec/support/sandboxing.rb:14:in `block (3 levels) in <top (required)>'
     # ./spec/support/sandboxing.rb:7:in `block (2 levels) in <top (required)>'

Finished in 11.92 seconds (files took 0.7278 seconds to load)
1803 examples, 1 failure, 1 pending

Failed examples:

rspec ./spec/integration/subprocess_exit_spec.rb:7 # When an example forks and exits it the subprocess prints the results only once

Randomized with seed 33805

I'm a bit at a loss for what's causing this. Anyone from @rspec/rspec got any ideas?

@@ -178,4 +178,12 @@ def self.const_missing(name)
require MODULES_TO_AUTOLOAD.fetch(name) { return super }
::RSpec.const_get(name)
end

module NonPassthroughExceptions
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

neat

@xaviershay
Copy link
Member

no idea ... if only we had a bisect feature... :P

@myronmarston
Copy link
Member Author

no idea ... if only we had a bisect feature... :P

Tried it. Here's the output so far:

➜  rspec-core git:(stop-rescuing-all-exceptions) bin/rspec --seed 33805 --bisect
Bisect started using options: "--seed 33805"
Running suite to find failures...

Bisect failed!

No failures found. Bisect only works in the presence of one or more failing examples.
➜  rspec-core git:(stop-rescuing-all-exceptions) bin/rspec --seed 33805 --bisect
Bisect started using options: "--seed 33805"
Running suite to find failures... (12.88 seconds)
Starting bisect with 1 failing example and 1802 non-failing examples.
Checking that failure(s) are order-dependent... failure appears to be order-dependent

Round 1: bisecting over non-failing examples 1-1802 .. multiple culprits detected - splitting candidates (10.75 seconds)
Round 2: bisecting over non-failing examples 1-901 .. multiple culprits detected - splitting candidates (11.29 seconds)
Round 3: bisecting over non-failing examples 1-451 .. multiple culprits detected - splitting candidates (15.52 seconds)
Round 4: bisecting over non-failing examples 1-226 .. multiple culprits detected - splitting candidates (17.66 seconds)
Round 5: bisecting over non-failing examples 1-113 .. multiple culprits detected - splitting candidates (18.34 seconds)
Round 6: bisecting over non-failing examples 1-57 .. multiple culprits detected - splitting candidates (18.98 seconds)
Round 7: bisecting over non-failing examples 1-29 .. multiple culprits detected - splitting candidates (21.38 seconds)
Round 8: bisecting over non-failing examples 1-15 .. multiple culprits detected - splitting candidates (19.44 seconds)
Round 9: bisecting over non-failing examples 1-8 .. multiple culprits detected - splitting candidates (19.54 seconds)
Round 10: bisecting over non-failing examples 1-4 .. multiple culprits detected - splitting candidates (19.26 seconds)
Round 11: bisecting over non-failing examples 1-2 .. multiple culprits detected - splitting candidates (19.19 seconds)
Round 12: bisecting over non-failing examples 3-4 .. multiple culprits detected - splitting candidates (19.4 seconds)
Round 13: bisecting over non-failing examples 5-8 .. multiple culprits detected - splitting candidates (20.88 seconds)
Round 14: bisecting over non-failing examples 5-6 .. multiple culprits detected - splitting candidates (19.49 seconds)
Round 15: bisecting over non-failing examples 7-8 .. multiple culprits detected - splitting candidates (18.9 seconds)
Round 16: bisecting over non-failing examples 9-15 .. multiple culprits detected - splitting candidates (20.57 seconds)
Round 17: bisecting over non-failing examples 9-12 .. multiple culprits detected - splitting candidates (20.12 seconds)
Round 18: bisecting over non-failing examples 9-10 .. multiple culprits detected - splitting candidates (19.71 seconds)
Round 19: bisecting over non-failing examples 11-12 .. multiple culprits detected - splitting candidates (19.88 seconds)
Round 20: bisecting over non-failing examples 13-15 .. multiple culprits detected - splitting candidates (19.26 seconds)
Round 21: bisecting over non-failing examples 13-14 .. multiple culprits detected - splitting candidates (20.12 seconds)
Round 22: bisecting over non-failing examples 16-29 .. multiple culprits detected - splitting candidates (19.43 seconds)
Round 23: bisecting over non-failing examples 16-22 .. multiple culprits detected - splitting candidates (21.4 seconds)
Round 24: bisecting over non-failing examples 16-19 .. multiple culprits detected - splitting candidates (20.82 seconds)
Round 25: bisecting over non-failing examples 16-17 .. multiple culprits detected - splitting candidates (20.64 seconds)
Round 26: bisecting over non-failing examples 18-19 .. multiple culprits detected - splitting candidates (20.58 seconds)
Round 27: bisecting over non-failing examples 20-22 .. multiple culprits detected - splitting candidates (19.98 seconds)
Round 28: bisecting over non-failing examples 20-21 .. multiple culprits detected - splitting candidates (20.84 seconds)
Round 29: bisecting over non-failing examples 23-29 .. multiple culprits detected - splitting candidates (18.94 seconds)
Round 30: bisecting over non-failing examples 23-26 .. multiple culprits detected - splitting candidates (19.09 seconds)
Round 31: bisecting over non-failing examples 23-24 .. multiple culprits detected - splitting candidates (21.05 seconds)
Round 32: bisecting over non-failing examples 25-26 .. multiple culprits detected - splitting candidates (19.08 seconds)
Round 33: bisecting over non-failing examples 27-29 .. multiple culprits detected - splitting candidates (18.98 seconds)
Round 34: bisecting over non-failing examples 27-28 .. multiple culprits detected - splitting candidates (19.34 seconds)
Round 35: bisecting over non-failing examples 30-57 .. multiple culprits detected - splitting candidates (19.26 seconds)
Round 36: bisecting over non-failing examples 30-43 .. multiple culprits detected - splitting candidates (18.55 seconds)
Round 37: bisecting over non-failing examples 30-36 .. multiple culprits detected - splitting candidates (18.51 seconds)
Round 38: bisecting over non-failing examples 30-33 .. multiple culprits detected - splitting candidates (18.66 seconds)
Round 39: bisecting over non-failing examples 30-31 .. multiple culprits detected - splitting candidates (18.65 seconds)
Round 40: bisecting over non-failing examples 32-33 .. multiple culprits detected - splitting candidates (18.71 seconds)
Round 41: bisecting over non-failing examples 34-36 .. multiple culprits detected - splitting candidates (18.67 seconds)
Round 42: bisecting over non-failing examples 34-35 . ignoring example 34 (9.29 seconds)
Round 43: bisecting over non-failing examples 37-43 .. multiple culprits detected - splitting candidates (18.62 seconds)
Round 44: bisecting over non-failing examples 37-40 .. multiple culprits detected - splitting candidates (18.75 seconds)
Round 45: bisecting over non-failing examples 37-38 .. multiple culprits detected - splitting candidates (20.05 seconds)
Round 46: bisecting over non-failing examples 39-40 .. multiple culprits detected - splitting candidates (19.6 seconds)
Round 47: bisecting over non-failing examples 41-43 .. multiple culprits detected - splitting candidates (19.88 seconds)
Round 48: bisecting over non-failing examples 41-42 .. multiple culprits detected - splitting candidates (19.8 seconds)
Round 49: bisecting over non-failing examples 44-57 .. multiple culprits detected - splitting candidates (19.11 seconds)
Round 50: bisecting over non-failing examples 44-50 .. multiple culprits detected - splitting candidates (19.83 seconds)
Round 51: bisecting over non-failing examples 44-47 .. multiple culprits detected - splitting candidates (19.79 seconds)
Round 52: bisecting over non-failing examples 44-45 .. multiple culprits detected - splitting candidates (19.15 seconds)
Round 53: bisecting over non-failing examples 46-47 .. multiple culprits detected - splitting candidates (18.73 seconds)
Round 54: bisecting over non-failing examples 48-50 .. multiple culprits detected - splitting candidates (18.7 seconds)
Round 55: bisecting over non-failing examples 48-49 .. multiple culprits detected - splitting candidates (20.07 seconds)
Round 56: bisecting over non-failing examples 51-57 .. multiple culprits detected - splitting candidates (19.41 seconds)
Round 57: bisecting over non-failing examples 51-54 .. multiple culprits detected - splitting candidates (19.35 seconds)
Round 58: bisecting over non-failing examples 51-52 .. multiple culprits detected - splitting candidates (19.79 seconds)
Round 59: bisecting over non-failing examples 53-54 .. multiple culprits detected - splitting candidates (21.11 seconds)
Round 60: bisecting over non-failing examples 55-57 .. multiple culprits detected - splitting candidates (19.62 seconds)
Round 61: bisecting over non-failing examples 55-56 .. multiple culprits detected - splitting candidates (19.15 seconds)
Round 62: bisecting over non-failing examples 58-113 .. multiple culprits detected - splitting candidates (18.95 seconds)
Round 63: bisecting over non-failing examples 58-85 .. multiple culprits detected - splitting candidates (19.36 seconds)
Round 64: bisecting over non-failing examples 58-71 .. multiple culprits detected - splitting candidates (20.03 seconds)
Round 65: bisecting over non-failing examples 58-64 .. multiple culprits detected - splitting candidates (19.76 seconds)
Round 66: bisecting over non-failing examples 58-61 .. multiple culprits detected - splitting candidates (20.53 seconds)
Round 67: bisecting over non-failing examples 58-59 .. multiple culprits detected - splitting candidates (22.56 seconds)
Round 68: bisecting over non-failing examples 60-61 .. multiple culprits detected - splitting candidates (21.94 seconds)
Round 69: bisecting over non-failing examples 62-64 .. multiple culprits detected - splitting candidates (20.35 seconds)
Round 70: bisecting over non-failing examples 62-63 .. multiple culprits detected - splitting candidates (19.1 seconds)
Round 71: bisecting over non-failing examples 65-71 .. multiple culprits detected - splitting candidates (19.78 seconds)
Round 72: bisecting over non-failing examples 65-68 .. multiple culprits detected - splitting candidates (18.57 seconds)
Round 73: bisecting over non-failing examples 65-66 .. multiple culprits detected - splitting candidates (20.76 seconds)
Round 74: bisecting over non-failing examples 67-68 .. multiple culprits detected - splitting candidates (29.03 seconds)
Round 75: bisecting over non-failing examples 69-71 .. multiple culprits detected - splitting candidates (26.06 seconds)
Round 76: bisecting over non-failing examples 69-70 .. multiple culprits detected - splitting candidates (23.39 seconds)
Round 77: bisecting over non-failing examples 72-85 .. multiple culprits detected - splitting candidates (20.34 seconds)
Round 78: bisecting over non-failing examples 72-78 .. multiple culprits detected - splitting candidates (18.84 seconds)
Round 79: bisecting over non-failing examples 72-75 .. multiple culprits detected - splitting candidates (18.84 seconds)
Round 80: bisecting over non-failing examples 72-73 .. multiple culprits detected - splitting candidates (23.25 seconds)
Round 81: bisecting over non-failing examples 74-75 .. multiple culprits detected - splitting candidates (23.63 seconds)
Round 82: bisecting over non-failing examples 76-78 .. multiple culprits detected - splitting candidates (21.37 seconds)
Round 83: bisecting over non-failing examples 76-77 .. multiple culprits detected - splitting candidates (19.62 seconds)
Round 84: bisecting over non-failing examples 79-85 .. multiple culprits detected - splitting candidates (22.92 seconds)
Round 85: bisecting over non-failing examples 79-82 .. multiple culprits detected - splitting candidates (22.75 seconds)
Round 86: bisecting over non-failing examples 79-80 .. multiple culprits detected - splitting candidates (22.88 seconds)
Round 87: bisecting over non-failing examples 81-82 .. multiple culprits detected - splitting candidates (22.77 seconds)
Round 88: bisecting over non-failing examples 83-85 .. multiple culprits detected - splitting candidates (22.91 seconds)
Round 89: bisecting over non-failing examples 83-84 .. multiple culprits detected - splitting candidates (23.08 seconds)
Round 90: bisecting over non-failing examples 86-113 .. multiple culprits detected - splitting candidates (24.14 seconds)
Round 91: bisecting over non-failing examples 86-99 .. multiple culprits detected - splitting candidates (22.4 seconds)
Round 92: bisecting over non-failing examples 86-92 .. multiple culprits detected - splitting candidates (22.46 seconds)
Round 93: bisecting over non-failing examples 86-89 .. multiple culprits detected - splitting candidates (22.56 seconds)
Round 94: bisecting over non-failing examples 86-87 .. multiple culprits detected - splitting candidates (22.43 seconds)
Round 95: bisecting over non-failing examples 88-89 .. multiple culprits detected - splitting candidates (22.84 seconds)
Round 96: bisecting over non-failing examples 90-92 .. multiple culprits detected - splitting candidates (22.42 seconds)
Round 97: bisecting over non-failing examples 90-91 .. multiple culprits detected - splitting candidates (22.58 seconds)
Round 98: bisecting over non-failing examples 93-99 .. multiple culprits detected - splitting candidates (21.54 seconds)
Round 99: bisecting over non-failing examples 93-96 .. multiple culprits detected - splitting candidates (22.64 seconds)
Round 100: bisecting over non-failing examples 93-94 .. multiple culprits detected - splitting candidates (22.33 seconds)
Round 101: bisecting over non-failing examples 95-96 .. multiple culprits detected - splitting candidates (22.43 seconds)
Round 102: bisecting over non-failing examples 97-99 .. multiple culprits detected - splitting candidates (21.58 seconds)
Round 103: bisecting over non-failing examples 97-98 .. multiple culprits detected - splitting candidates (19.24 seconds)
Round 104: bisecting over non-failing examples 100-113 .. multiple culprits detected - splitting candidates (18.68 seconds)
Round 105: bisecting over non-failing examples 100-106 .. multiple culprits detected - splitting candidates (18.7 seconds)
Round 106: bisecting over non-failing examples 100-103 .. multiple culprits detected - splitting candidates (22.92 seconds)
Round 107: bisecting over non-failing examples 100-101 .. multiple culprits detected - splitting candidates (22.46 seconds)
Round 108: bisecting over non-failing examples 102-103 .. multiple culprits detected - splitting candidates (22.63 seconds)
Round 109: bisecting over non-failing examples 104-106 .. multiple culprits detected - splitting candidates (22.64 seconds)
Round 110: bisecting over non-failing examples 104-105 .. multiple culprits detected - splitting candidates (22.7 seconds)
Round 111: bisecting over non-failing examples 107-113 .. multiple culprits detected - splitting candidates (22.59 seconds)
Round 112: bisecting over non-failing examples 107-110 .. multiple culprits detected - splitting candidates (22.54 seconds)
Round 113: bisecting over non-failing examples 107-108 .. multiple culprits detected - splitting candidates (21.88 seconds)
Round 114: bisecting over non-failing examples 109-110 .. multiple culprits detected - splitting candidates (19.1 seconds)
Round 115: bisecting over non-failing examples 111-113 .. multiple culprits detected - splitting candidates (22.1 seconds)
Round 116: bisecting over non-failing examples 111-112 .. multiple culprits detected - splitting candidates (23.24 seconds)
Round 117: bisecting over non-failing examples 114-226 .. multiple culprits detected - splitting candidates (22.48 seconds)
Round 118: bisecting over non-failing examples 114-170 .. multiple culprits detected - splitting candidates (22.87 seconds)
Round 119: bisecting over non-failing examples 114-142 .. multiple culprits detected - splitting candidates (22.81 seconds)
Round 120: bisecting over non-failing examples 114-128 .. multiple culprits detected - splitting candidates (20.63 seconds)
Round 121: bisecting over non-failing examples 114-121 .. multiple culprits detected - splitting candidates (20.45 seconds)
Round 122: bisecting over non-failing examples 114-117 .. multiple culprits detected - splitting candidates (22.88 seconds)
Round 123: bisecting over non-failing examples 114-115 .. multiple culprits detected - splitting candidates (22.68 seconds)
Round 124: bisecting over non-failing examples 116-117 .. multiple culprits detected - splitting candidates (19.22 seconds)
Round 125: bisecting over non-failing examples 118-121 .. multiple culprits detected - splitting candidates (22.32 seconds)
Round 126: bisecting over non-failing examples 118-119 .. multiple culprits detected - splitting candidates (23.11 seconds)
Round 127: bisecting over non-failing examples 120-121 .. multiple culprits detected - splitting candidates (22.76 seconds)
Round 128: bisecting over non-failing examples 122-128 .. multiple culprits detected - splitting candidates (22.71 seconds)
Round 129: bisecting over non-failing examples 122-125 .. multiple culprits detected - splitting candidates (20.6 seconds)
Round 130: bisecting over non-failing examples 122-123 .. multiple culprits detected - splitting candidates (18.72 seconds)
Round 131: bisecting over non-failing examples 124-125 .. multiple culprits detected - splitting candidates (20.36 seconds)
Round 132: bisecting over non-failing examples 126-128 .. multiple culprits detected - splitting candidates (21.33 seconds)
Round 133: bisecting over non-failing examples 126-127 .. multiple culprits detected - splitting candidates (22.75 seconds)
Round 134: bisecting over non-failing examples 129-142 .. multiple culprits detected - splitting candidates (20.74 seconds)
Round 135: bisecting over non-failing examples 129-135 .. multiple culprits detected - splitting candidates (20.9 seconds)
Round 136: bisecting over non-failing examples 129-132 .. multiple culprits detected - splitting candidates (22.2 seconds)
Round 137: bisecting over non-failing examples 129-130 .. multiple culprits detected - splitting candidates (23.74 seconds)
Round 138: bisecting over non-failing examples 131-132 .. multiple culprits detected - splitting candidates (22.53 seconds)
Round 139: bisecting over non-failing examples 133-135 .. multiple culprits detected - splitting candidates (21.86 seconds)
Round 140: bisecting over non-failing examples 133-134 .. multiple culprits detected - splitting candidates (22.42 seconds)
Round 141: bisecting over non-failing examples 136-142 .. multiple culprits detected - splitting candidates (22.21 seconds)
Round 142: bisecting over non-failing examples 136-139 .. multiple culprits detected - splitting candidates (22.31 seconds)
Round 143: bisecting over non-failing examples 136-137 .. multiple culprits detected - splitting candidates (22.31 seconds)
Round 144: bisecting over non-failing examples 138-139 .. multiple culprits detected - splitting candidates (22.53 seconds)
Round 145: bisecting over non-failing examples 140-142 .. multiple culprits detected - splitting candidates (21.24 seconds)
Round 146: bisecting over non-failing examples 140-141 .. multiple culprits detected - splitting candidates (22.47 seconds)
Round 147: bisecting over non-failing examples 143-170 .. multiple culprits detected - splitting candidates (23.95 seconds)
Round 148: bisecting over non-failing examples 143-156 .. multiple culprits detected - splitting candidates (23.81 seconds)
Round 149: bisecting over non-failing examples 143-149 .. multiple culprits detected - splitting candidates (23.69 seconds)
Round 150: bisecting over non-failing examples 143-146 .. multiple culprits detected - splitting candidates (23.96 seconds)
Round 151: bisecting over non-failing examples 143-144 .. multiple culprits detected - splitting candidates (22.51 seconds)
Round 152: bisecting over non-failing examples 145-146 .. multiple culprits detected - splitting candidates (22.58 seconds)
Round 153: bisecting over non-failing examples 147-149 .. multiple culprits detected - splitting candidates (22.38 seconds)
Round 154: bisecting over non-failing examples 147-148 .. multiple culprits detected - splitting candidates (22.29 seconds)
Round 155: bisecting over non-failing examples 150-156 .. multiple culprits detected - splitting candidates (21.1 seconds)
Round 156: bisecting over non-failing examples 150-153 .. multiple culprits detected - splitting candidates (6 minutes 26 seconds)
Round 157: bisecting over non-failing examples 150-151 .. multiple culprits detected - splitting candidates (57 minutes 52 seconds)
Round 158: bisecting over non-failing examples 152-153 .. multiple culprits detected - splitting candidates (23.02 seconds)
Round 159: bisecting over non-failing examples 154-156 .. multiple culprits detected - splitting candidates (22.52 seconds)
Round 160: bisecting over non-failing examples 154-155 .. multiple culprits detected - splitting candidates (23.9 seconds)
Round 161: bisecting over non-failing examples 157-170
.


. multiple culprits detected - splitting candidates (22.31 seconds)
Round 162: bisecting over non-failing examples 157-163 .. multiple culprits detected - splitting candidates (28.32 seconds)
Round 163: bisecting over non-failing examples 157-160 .. multiple culprits detected - splitting candidates (21.67 seconds)
Round 164: bisecting over non-failing examples 157-158 .. multiple culprits detected - splitting candidates (21.25 seconds)
Round 165: bisecting over non-failing examples 159-160 .. multiple culprits detected - splitting candidates (21.42 seconds)
Round 166: bisecting over non-failing examples 161-163 .. multiple culprits detected - splitting candidates (21.08 seconds)
Round 167: bisecting over non-failing examples 161-162 .. multiple culprits detected - splitting candidates (20.8 seconds)
Round 168: bisecting over non-failing examples 164-170 .. multiple culprits detected - splitting candidates (21.25 seconds)
Round 169: bisecting over non-failing examples 164-167 .. multiple culprits detected - splitting candidates (22.37 seconds)
Round 170: bisecting over non-failing examples 164-165 .. multiple culprits detected - splitting candidates (21.04 seconds)
Round 171: bisecting over non-failing examples 166-167 .. multiple culprits detected - splitting candidates (20.71 seconds)
Round 172: bisecting over non-failing examples 168-170 .. multiple culprits detected - splitting candidates (20.62 seconds)
Round 173: bisecting over non-failing examples 168-169 .. multiple culprits detected - splitting candidates (20.51 seconds)
Round 174: bisecting over non-failing examples 171-226 .. multiple culprits detected - splitting candidates (20.09 seconds)
Round 175: bisecting over non-failing examples 171-198 .. multiple culprits detected - splitting candidates (20.49 seconds)
Round 176: bisecting over non-failing examples 171-184 .. multiple culprits detected - splitting candidates (20.41 seconds)
Round 177: bisecting over non-failing examples 171-177 .. multiple culprits detected - splitting candidates (27.12 seconds)
Round 178: bisecting over non-failing examples 171-174 .. multiple culprits detected - splitting candidates (24.93 seconds)
Round 179: bisecting over non-failing examples 171-172 .. multiple culprits detected - splitting candidates (20.4 seconds)
Round 180: bisecting over non-failing examples 173-174 .. multiple culprits detected - splitting candidates (20.5 seconds)
Round 181: bisecting over non-failing examples 175-177 .. multiple culprits detected - splitting candidates (20.11 seconds)
Round 182: bisecting over non-failing examples 175-176 .. multiple culprits detected - splitting candidates (20.47 seconds)
Round 183: bisecting over non-failing examples 178-184 .. multiple culprits detected - splitting candidates (21.97 seconds)
Round 184: bisecting over non-failing examples 178-181 .. multiple culprits detected - splitting candidates (22.92 seconds)
Round 185: bisecting over non-failing examples 178-179 .. multiple culprits detected - splitting candidates (22.63 seconds)
Round 186: bisecting over non-failing examples 180-181 .. multiple culprits detected - splitting candidates (25.49 seconds)
Round 187: bisecting over non-failing examples 182-184 .. multiple culprits detected - splitting candidates (23.14 seconds)
Round 188: bisecting over non-failing examples 182-183 .. multiple culprits detected - splitting candidates (23.77 seconds)
Round 189: bisecting over non-failing examples 185-198 .. ignoring examples 192-198 (31.51 seconds)
Round 190: bisecting over non-failing examples 185-191 .. multiple culprits detected - splitting candidates (32.45 seconds)
Round 191: bisecting over non-failing examples 185-188 .. multiple culprits detected - splitting candidates (23.96 seconds)
Round 192: bisecting over non-failing examples 185-186 .. multiple culprits detected - splitting candidates (19.68 seconds)
Round 193: bisecting over non-failing examples 187-188 .. multiple culprits detected - splitting candidates (20.92 seconds)
Round 194: bisecting over non-failing examples 189-191 .. multiple culprits detected - splitting candidates (21.63 seconds)
Round 195: bisecting over non-failing examples 189-190 .. multiple culprits detected - splitting candidates (19.84 seconds)
Round 196: bisecting over non-failing examples 199-226 .. multiple culprits detected - splitting candidates (21.26 seconds)
Round 197: bisecting over non-failing examples 199-212 .. multiple culprits detected - splitting candidates (20.6 seconds)
Round 198: bisecting over non-failing examples 199-205 .. multiple culprits detected - splitting candidates (19.98 seconds)
Round 199: bisecting over non-failing examples 199-202 .. multiple culprits detected - splitting candidates (20.41 seconds)
Round 200: bisecting over non-failing examples 199-200 .. multiple culprits detected - splitting candidates (20.09 seconds)
Round 201: bisecting over non-failing examples 201-202 .. multiple culprits detected - splitting candidates (19.58 seconds)
Round 202: bisecting over non-failing examples 203-205 .. multiple culprits detected - splitting candidates (20.22 seconds)
Round 203: bisecting over non-failing examples 203-204 .. multiple culprits detected - splitting candidates (21.12 seconds)
Round 204: bisecting over non-failing examples 206-212 .. multiple culprits detected - splitting candidates (21.28 seconds)
Round 205: bisecting over non-failing examples 206-209 .. multiple culprits detected - splitting candidates (21.64 seconds)
Round 206: bisecting over non-failing examples 206-207 .. multiple culprits detected - splitting candidates (20.77 seconds)
Round 207: bisecting over non-failing examples 208-209 .. multiple culprits detected - splitting candidates (21.06 seconds)
Round 208: bisecting over non-failing examples 210-212 .. multiple culprits detected - splitting candidates (20.92 seconds)
Round 209: bisecting over non-failing examples 210-211 .. multiple culprits detected - splitting candidates (20.77 seconds)
Round 210: bisecting over non-failing examples 213-226 .. multiple culprits detected - splitting candidates (20.68 seconds)
Round 211: bisecting over non-failing examples 213-219 .. multiple culprits detected - splitting candidates (21.49 seconds)
Round 212: bisecting over non-failing examples 213-216 .. multiple culprits detected - splitting candidates (20.37 seconds)
Round 213: bisecting over non-failing examples 213-214 .. multiple culprits detected - splitting candidates (21.43 seconds)
Round 214: bisecting over non-failing examples 215-216 .. multiple culprits detected - splitting candidates (20.9 seconds)
Round 215: bisecting over non-failing examples 217-219 .. multiple culprits detected - splitting candidates (21.59 seconds)
Round 216: bisecting over non-failing examples 217-218 .. multiple culprits detected - splitting candidates (20.45 seconds)
Round 217: bisecting over non-failing examples 220-226 .. multiple culprits detected - splitting candidates (20.55 seconds)
Round 218: bisecting over non-failing examples 220-223 .. multiple culprits detected - splitting candidates (21.37 seconds)
Round 219: bisecting over non-failing examples 220-221 .. multiple culprits detected - splitting candidates (21.43 seconds)
Round 220: bisecting over non-failing examples 222-223 .. multiple culprits detected - splitting candidates (20.23 seconds)
Round 221: bisecting over non-failing examples 224-226 .. multiple culprits detected - splitting candidates (22.13 seconds)
Round 222: bisecting over non-failing examples 224-225 .. multiple culprits detected - splitting candidates (21.7 seconds)
Round 223: bisecting over non-failing examples 227-451 .. multiple culprits detected - splitting candidates (18.27 seconds)
Round 224: bisecting over non-failing examples 227-339 .. multiple culprits detected - splitting candidates (20.33 seconds)
Round 225: bisecting over non-failing examples 227-283 .. multiple culprits detected - splitting candidates (20.29 seconds)
Round 226: bisecting over non-failing examples 227-255 .. multiple culprits detected - splitting candidates (20.01 seconds)
Round 227: bisecting over non-failing examples 227-241 .. multiple culprits detected - splitting candidates (20.6 seconds)
Round 228: bisecting over non-failing examples 227-234 .. multiple culprits detected - splitting candidates (20.31 seconds)
Round 229: bisecting over non-failing examples 227-230 .. multiple culprits detected - splitting candidates (21.79 seconds)
Round 230: bisecting over non-failing examples 227-228 .. multiple culprits detected - splitting candidates (21.8 seconds)
Round 231: bisecting over non-failing examples 229-230 .. multiple culprits detected - splitting candidates (20.78 seconds)
Round 232: bisecting over non-failing examples 231-234 .. multiple culprits detected - splitting candidates (20.97 seconds)
Round 233: bisecting over non-failing examples 231-232 .. multiple culprits detected - splitting candidates (22.09 seconds)
Round 234: bisecting over non-failing examples 233-234 .. multiple culprits detected - splitting candidates (21.27 seconds)
Round 235: bisecting over non-failing examples 235-241 .. multiple culprits detected - splitting candidates (20.22 seconds)
Round 236: bisecting over non-failing examples 235-238 .. multiple culprits detected - splitting candidates (21.08 seconds)
Round 237: bisecting over non-failing examples 235-236 .. multiple culprits detected - splitting candidates (20.2 seconds)
Round 238: bisecting over non-failing examples 237-238 .. multiple culprits detected - splitting candidates (20.24 seconds)
Round 239: bisecting over non-failing examples 239-241 .. multiple culprits detected - splitting candidates (21.21 seconds)
Round 240: bisecting over non-failing examples 239-240 .. multiple culprits detected - splitting candidates (21.99 seconds)
Round 241: bisecting over non-failing examples 242-255 .. multiple culprits detected - splitting candidates (22.13 seconds)
Round 242: bisecting over non-failing examples 242-248 .. multiple culprits detected - splitting candidates (22.03 seconds)
Round 243: bisecting over non-failing examples 242-245 .. multiple culprits detected - splitting candidates (20.67 seconds)
Round 244: bisecting over non-failing examples 242-243 .. multiple culprits detected - splitting candidates (21.47 seconds)
Round 245: bisecting over non-failing examples 244-245 .. multiple culprits detected - splitting candidates (20.69 seconds)
Round 246: bisecting over non-failing examples 246-248 .. multiple culprits detected - splitting candidates (21.34 seconds)
Round 247: bisecting over non-failing examples 246-247 .. multiple culprits detected - splitting candidates (21.57 seconds)
Round 248: bisecting over non-failing examples 249-255 .. multiple culprits detected - splitting candidates (21.65 seconds)
Round 249: bisecting over non-failing examples 249-252 .. multiple culprits detected - splitting candidates (20.41 seconds)
Round 250: bisecting over non-failing examples 249-250 .. multiple culprits detected - splitting candidates (21.6 seconds)
Round 251: bisecting over non-failing examples 251-252 .. multiple culprits detected - splitting candidates (18.69 seconds)
Round 252: bisecting over non-failing examples 253-255 .. ignoring example 255 (19 seconds)
Round 253: bisecting over non-failing examples 253-254 .. multiple culprits detected - splitting candidates (19.24 seconds)
Round 254: bisecting over non-failing examples 256-283 .. multiple culprits detected - splitting candidates (19.24 seconds)
Round 255: bisecting over non-failing examples 256-269 .. multiple culprits detected - splitting candidates (19.42 seconds)
Round 256: bisecting over non-failing examples 256-262 .. multiple culprits detected - splitting candidates (19.11 seconds)
Round 257: bisecting over non-failing examples 256-259 .. multiple culprits detected - splitting candidates (19.32 seconds)
Round 258: bisecting over non-failing examples 256-257 .. multiple culprits detected - splitting candidates (19.29 seconds)
Round 259: bisecting over non-failing examples 258-259 .. multiple culprits detected - splitting candidates (19.09 seconds)
Round 260: bisecting over non-failing examples 260-262 .. multiple culprits detected - splitting candidates (19.02 seconds)
Round 261: bisecting over non-failing examples 260-261 .. multiple culprits detected - splitting candidates (18.77 seconds)
Round 262: bisecting over non-failing examples 263-269 .. multiple culprits detected - splitting candidates (18.86 seconds)
Round 263: bisecting over non-failing examples 263-266 .. multiple culprits detected - splitting candidates (18.89 seconds)
Round 264: bisecting over non-failing examples 263-264 .. multiple culprits detected - splitting candidates (19.71 seconds)
Round 265: bisecting over non-failing examples 265-266 .. multiple culprits detected - splitting candidates (18.95 seconds)
Round 266: bisecting over non-failing examples 267-269 .. multiple culprits detected - splitting candidates (18.94 seconds)
Round 267: bisecting over non-failing examples 267-268 .. multiple culprits detected - splitting candidates (18.93 seconds)
Round 268: bisecting over non-failing examples 270-283 .. multiple culprits detected - splitting candidates (18.78 seconds)
Round 269: bisecting over non-failing examples 270-276 .. multiple culprits detected - splitting candidates (18.91 seconds)
Round 270: bisecting over non-failing examples 270-273 .. multiple culprits detected - splitting candidates (19.18 seconds)
Round 271: bisecting over non-failing examples 270-271 .. multiple culprits detected - splitting candidates (18.88 seconds)
Round 272: bisecting over non-failing examples 272-273 .. multiple culprits detected - splitting candidates (18.98 seconds)
Round 273: bisecting over non-failing examples 274-276 .. multiple culprits detected - splitting candidates (18.94 seconds)
Round 274: bisecting over non-failing examples 274-275 .. ignoring example 275 (19.04 seconds)
Round 275: bisecting over non-failing examples 277-283 .. multiple culprits detected - splitting candidates (19.07 seconds)
Round 276: bisecting over non-failing examples 277-280 .. multiple culprits detected - splitting candidates (18.88 seconds)
Round 277: bisecting over non-failing examples 277-278 .. multiple culprits detected - splitting candidates (19.21 seconds)
Round 278: bisecting over non-failing examples 279-280 .. multiple culprits detected - splitting candidates (18.97 seconds)
Round 279: bisecting over non-failing examples 281-283 .. multiple culprits detected - splitting candidates (19.12 seconds)
Round 280: bisecting over non-failing examples 281-282 .. multiple culprits detected - splitting candidates (18.8 seconds)
Round 281: bisecting over non-failing examples 284-339 .. multiple culprits detected - splitting candidates (19.31 seconds)
Round 282: bisecting over non-failing examples 284-311 .. multiple culprits detected - splitting candidates (18.93 seconds)
Round 283: bisecting over non-failing examples 284-297 .. multiple culprits detected - splitting candidates (19.21 seconds)
Round 284: bisecting over non-failing examples 284-290 .. multiple culprits detected - splitting candidates (19.32 seconds)
Round 285: bisecting over non-failing examples 284-287 .. multiple culprits detected - splitting candidates (19.38 seconds)
Round 286: bisecting over non-failing examples 284-285 .. multiple culprits detected - splitting candidates (19.26 seconds)
Round 287: bisecting over non-failing examples 286-287 . ignoring example 286 (9.56 seconds)
Round 288: bisecting over non-failing examples 288-290 .. multiple culprits detected - splitting candidates (19.65 seconds)
Round 289: bisecting over non-failing examples 288-289 .. multiple culprits detected - splitting candidates (20.49 seconds)
Round 290: bisecting over non-failing examples 291-297 .. multiple culprits detected - splitting candidates (18.77 seconds)
Round 291: bisecting over non-failing examples 291-294 .. multiple culprits detected - splitting candidates (18.94 seconds)
Round 292: bisecting over non-failing examples 291-292 .. multiple culprits detected - splitting candidates (19.09 seconds)
Round 293: bisecting over non-failing examples 293-294 .. multiple culprits detected - splitting candidates (19.1 seconds)
Round 294: bisecting over non-failing examples 295-297 .. multiple culprits detected - splitting candidates (19.23 seconds)
Round 295: bisecting over non-failing examples 295-296 .. multiple culprits detected - splitting candidates (19.85 seconds)
Round 296: bisecting over non-failing examples 298-311 .. multiple culprits detected - splitting candidates (23.54 seconds)
Round 297: bisecting over non-failing examples 298-304 .. multiple culprits detected - splitting candidates (25.12 seconds)
Round 298: bisecting over non-failing examples 298-301 .. multiple culprits detected - splitting candidates (20.34 seconds)
Round 299: bisecting over non-failing examples 298-299 .. multiple culprits detected - splitting candidates (21.55 seconds)
Round 300: bisecting over non-failing examples 300-301 .. multiple culprits detected - splitting candidates (19.2 seconds)
Round 301: bisecting over non-failing examples 302-304 .. multiple culprits detected - splitting candidates (19.51 seconds)
Round 302: bisecting over non-failing examples 302-303 .. multiple culprits detected - splitting candidates (23.64 seconds)
Round 303: bisecting over non-failing examples 305-311 .. multiple culprits detected - splitting candidates (23.17 seconds)
Round 304: bisecting over non-failing examples 305-308 .. multiple culprits detected - splitting candidates (19.49 seconds)
Round 305: bisecting over non-failing examples 305-306 .. multiple culprits detected - splitting candidates (19.79 seconds)
Round 306: bisecting over non-failing examples 307-308 .. multiple culprits detected - splitting candidates (19.37 seconds)
Round 307: bisecting over non-failing examples 309-311 .. multiple culprits detected - splitting candidates (19.72 seconds)
Round 308: bisecting over non-failing examples 309-310 .. multiple culprits detected - splitting candidates (22.37 seconds)
Round 309: bisecting over non-failing examples 312-339 .. multiple culprits detected - splitting candidates (19.83 seconds)
Round 310: bisecting over non-failing examples 312-325 .. multiple culprits detected - splitting candidates (20.51 seconds)
Round 311: bisecting over non-failing examples 312-318 .. multiple culprits detected - splitting candidates (20.11 seconds)
Round 312: bisecting over non-failing examples 312-315 .. multiple culprits detected - splitting candidates (21 seconds)
Round 313: bisecting over non-failing examples 312-313 .. multiple culprits detected - splitting candidates (20.49 seconds)
Round 314: bisecting over non-failing examples 314-315 .. multiple culprits detected - splitting candidates (19.83 seconds)
Round 315: bisecting over non-failing examples 316-318 .. multiple culprits detected - splitting candidates (20.98 seconds)
Round 316: bisecting over non-failing examples 316-317 .. multiple culprits detected - splitting candidates (20.68 seconds)
Round 317: bisecting over non-failing examples 319-325 .. multiple culprits detected - splitting candidates (20.89 seconds)
Round 318: bisecting over non-failing examples 319-322 .. multiple culprits detected - splitting candidates (20.11 seconds)
Round 319: bisecting over non-failing examples 319-320 .. multiple culprits detected - splitting candidates (19.87 seconds)
Round 320: bisecting over non-failing examples 321-322 .. multiple culprits detected - splitting candidates (19.47 seconds)
Round 321: bisecting over non-failing examples 323-325 .. multiple culprits detected - splitting candidates (19.59 seconds)
Round 322: bisecting over non-failing examples 323-324 .. multiple culprits detected - splitting candidates (20.16 seconds)
Round 323: bisecting over non-failing examples 326-339 .. multiple culprits detected - splitting candidates (20 seconds)
Round 324: bisecting over non-failing examples 326-332 .. ignoring examples 330-332 (22.81 seconds)
Round 325: bisecting over non-failing examples 326-329 .. multiple culprits detected - splitting candidates (25.68 seconds)
Round 326: bisecting over non-failing examples 326-327 .. multiple culprits detected - splitting candidates (22.19 seconds)
Round 327: bisecting over non-failing examples 328-329 .. multiple culprits detected - splitting candidates (21.18 seconds)
Round 328: bisecting over non-failing examples 333-339 .. multiple culprits detected - splitting candidates (20.75 seconds)
Round 329: bisecting over non-failing examples 333-336 .. multiple culprits detected - splitting candidates (20.61 seconds)
Round 330: bisecting over non-failing examples 333-334 .. multiple culprits detected - splitting candidates (20.53 seconds)
Round 331: bisecting over non-failing examples 335-336 .. multiple culprits detected - splitting candidates (20.75 seconds)
Round 332: bisecting over non-failing examples 337-339 .. multiple culprits detected - splitting candidates (22.09 seconds)
Round 333: bisecting over non-failing examples 337-338 .. multiple culprits detected - splitting candidates (20.07 seconds)
Round 334: bisecting over non-failing examples 340-451 .. multiple culprits detected - splitting candidates (19.21 seconds)
Round 335: bisecting over non-failing examples 340-395 .. multiple culprits detected - splitting candidates (19.68 seconds)
Round 336: bisecting over non-failing examples 340-367 .. multiple culprits detected - splitting candidates (19.9 seconds)
Round 337: bisecting over non-failing examples 340-353 .. multiple culprits detected - splitting candidates (20.4 seconds)
Round 338: bisecting over non-failing examples 340-346 .. multiple culprits detected - splitting candidates (19.85 seconds)
Round 339: bisecting over non-failing examples 340-343 .. multiple culprits detected - splitting candidates (19.98 seconds)
Round 340: bisecting over non-failing examples 340-341 .. multiple culprits detected - splitting candidates (22.29 seconds)
Round 341: bisecting over non-failing examples 342-343 .. multiple culprits detected - splitting candidates (21.2 seconds)
Round 342: bisecting over non-failing examples 344-346 .. multiple culprits detected - splitting candidates (21.83 seconds)
Round 343: bisecting over non-failing examples 344-345 .. multiple culprits detected - splitting candidates (19.68 seconds)
Round 344: bisecting over non-failing examples 347-353 .. multiple culprits detected - splitting candidates (19.7 seconds)
Round 345: bisecting over non-failing examples 347-350 .. multiple culprits detected - splitting candidates (21.18 seconds)
Round 346: bisecting over non-failing examples 347-348 .. multiple culprits detected - splitting candidates (19.33 seconds)
Round 347: bisecting over non-failing examples 349-350 .. multiple culprits detected - splitting candidates (19.11 seconds)
Round 348: bisecting over non-failing examples 351-353 .. multiple culprits detected - splitting candidates (19.57 seconds)
Round 349: bisecting over non-failing examples 351-352 .. multiple culprits detected - splitting candidates (20.88 seconds)
Round 350: bisecting over non-failing examples 354-367 .. multiple culprits detected - splitting candidates (27.56 seconds)
Round 351: bisecting over non-failing examples 354-360 .. multiple culprits detected - splitting candidates (29.69 seconds)
Round 352: bisecting over non-failing examples 354-357 .. multiple culprits detected - splitting candidates (20.6 seconds)
Round 353: bisecting over non-failing examples 354-355 .. multiple culprits detected - splitting candidates (20.95 seconds)
Round 354: bisecting over non-failing examples 356-357 .. multiple culprits detected - splitting candidates (19.36 seconds)
Round 355: bisecting over non-failing examples 358-360 .. multiple culprits detected - splitting candidates (24.35 seconds)
Round 356: bisecting over non-failing examples 358-359 .. multiple culprits detected - splitting candidates (21.45 seconds)
Round 357: bisecting over non-failing examples 361-367 .. multiple culprits detected - splitting candidates (20.41 seconds)
Round 358: bisecting over non-failing examples 361-364 .. multiple culprits detected - splitting candidates (21.07 seconds)
Round 359: bisecting over non-failing examples 361-362 .. multiple culprits detected - splitting candidates (21.36 seconds)
Round 360: bisecting over non-failing examples 363-364 .. multiple culprits detected - splitting candidates (21.18 seconds)
Round 361: bisecting over non-failing examples 365-367 .. multiple culprits detected - splitting candidates (23.16 seconds)
Round 362: bisecting over non-failing examples 365-366 .. multiple culprits detected - splitting candidates (20.98 seconds)
Round 363: bisecting over non-failing examples 368-395 .. multiple culprits detected - splitting candidates (20.83 seconds)
Round 364: bisecting over non-failing examples 368-381 .. multiple culprits detected - splitting candidates (20.66 seconds)
Round 365: bisecting over non-failing examples 368-374 .. multiple culprits detected - splitting candidates (20.34 seconds)
Round 366: bisecting over non-failing examples 368-371 .. multiple culprits detected - splitting candidates (19.75 seconds)
Round 367: bisecting over non-failing examples 368-369 .. multiple culprits detected - splitting candidates (33.61 seconds)
Round 368: bisecting over non-failing examples 370-371 .. multiple culprits detected - splitting candidates (35.33 seconds)
Round 369: bisecting over non-failing examples 372-374 .. multiple culprits detected - splitting candidates (26.6 seconds)
Round 370: bisecting over non-failing examples 372-373 .. multiple culprits detected - splitting candidates (23.41 seconds)
Round 371: bisecting over non-failing examples 375-381 .. multiple culprits detected - splitting candidates (21.13 seconds)
Round 372: bisecting over non-failing examples 375-378 .. multiple culprits detected - splitting candidates (20.77 seconds)
Round 373: bisecting over non-failing examples 375-376 .. multiple culprits detected - splitting candidates (19.38 seconds)
Round 374: bisecting over non-failing examples 377-378 .. multiple culprits detected - splitting candidates (20.58 seconds)
Round 375: bisecting over non-failing examples 379-381 .. multiple culprits detected - splitting candidates (20.52 seconds)
Round 376: bisecting over non-failing examples 379-380 .. multiple culprits detected - splitting candidates (19.04 seconds)
Round 377: bisecting over non-failing examples 382-395 .. multiple culprits detected - splitting candidates (20.2 seconds)
Round 378: bisecting over non-failing examples 382-388 .. multiple culprits detected - splitting candidates (21.59 seconds)
Round 379: bisecting over non-failing examples 382-385 .. multiple culprits detected - splitting candidates (22.7 seconds)
Round 380: bisecting over non-failing examples 382-383 .. multiple culprits detected - splitting candidates (21.36 seconds)
Round 381: bisecting over non-failing examples 384-385 .. multiple culprits detected - splitting candidates (19.36 seconds)
Round 382: bisecting over non-failing examples 386-388 .. multiple culprits detected - splitting candidates (20.9 seconds)
Round 383: bisecting over non-failing examples 386-387 .. multiple culprits detected - splitting candidates (21.06 seconds)
Round 384: bisecting over non-failing examples 389-395 .. multiple culprits detected - splitting candidates (19.28 seconds)
Round 385: bisecting over non-failing examples 389-392 .. multiple culprits detected - splitting candidates (19.6 seconds)
Round 386: bisecting over non-failing examples 389-390 .. multiple culprits detected - splitting candidates (23.07 seconds)
Round 387: bisecting over non-failing examples 391-392 .. multiple culprits detected - splitting candidates (20.12 seconds)
Round 388: bisecting over non-failing examples 393-395 .. multiple culprits detected - splitting candidates (19.23 seconds)
Round 389: bisecting over non-failing examples 393-394 .. multiple culprits detected - splitting candidates (19.39 seconds)
Round 390: bisecting over non-failing examples 396-451 .. multiple culprits detected - splitting candidates (17.25 seconds)
Round 391: bisecting over non-failing examples 396-423 .. multiple culprits detected - splitting candidates (18.57 seconds)
Round 392: bisecting over non-failing examples 396-409 .. multiple culprits detected - splitting candidates (18.8 seconds)
Round 393: bisecting over non-failing examples 396-402 .. multiple culprits detected - splitting candidates (18.79 seconds)
Round 394: bisecting over non-failing examples 396-399 .. multiple culprits detected - splitting candidates (18.65 seconds)
Round 395: bisecting over non-failing examples 396-397 .. multiple culprits detected - splitting candidates (18.87 seconds)
Round 396: bisecting over non-failing examples 398-399 .. multiple culprits detected - splitting candidates (18.83 seconds)
Round 397: bisecting over non-failing examples 400-402 .. multiple culprits detected - splitting candidates (18.85 seconds)
Round 398: bisecting over non-failing examples 400-401 .. multiple culprits detected - splitting candidates (18.75 seconds)
Round 399: bisecting over non-failing examples 403-409 .. multiple culprits detected - splitting candidates (18.85 seconds)
Round 400: bisecting over non-failing examples 403-406 .. multiple culprits detected - splitting candidates (19.92 seconds)
Round 401: bisecting over non-failing examples 403-404 .. multiple culprits detected - splitting candidates (19.55 seconds)
Round 402: bisecting over non-failing examples 405-406 .. multiple culprits detected - splitting candidates (21.09 seconds)
Round 403: bisecting over non-failing examples 407-409 .. multiple culprits detected - splitting candidates (21.77 seconds)
Round 404: bisecting over non-failing examples 407-408 .. multiple culprits detected - splitting candidates (20.93 seconds)
Round 405: bisecting over non-failing examples 410-423 .. multiple culprits detected - splitting candidates (22.09 seconds)
Round 406: bisecting over non-failing examples 410-416 .. multiple culprits detected - splitting candidates (30.22 seconds)
Round 407: bisecting over non-failing examples 410-413 .. multiple culprits detected - splitting candidates (26.08 seconds)
Round 408: bisecting over non-failing examples 410-411 .. multiple culprits detected - splitting candidates (22.35 seconds)
Round 409: bisecting over non-failing examples 412-413 .. multiple culprits detected - splitting candidates (23.34 seconds)
Round 410: bisecting over non-failing examples 414-416 .. multiple culprits detected - splitting candidates (25.84 seconds)
Round 411: bisecting over non-failing examples 414-415 .. multiple culprits detected - splitting candidates (23.05 seconds)
Round 412: bisecting over non-failing examples 417-423 .. multiple culprits detected - splitting candidates (21.89 seconds)
Round 413: bisecting over non-failing examples 417-420 .. multiple culprits detected - splitting candidates (21.93 seconds)
Round 414: bisecting over non-failing examples 417-418 .. multiple culprits detected - splitting candidates (21.93 seconds)
Round 415: bisecting over non-failing examples 419-420 .. multiple culprits detected - splitting candidates (23.95 seconds)
Round 416: bisecting over non-failing examples 421-423 .. multiple culprits detected - splitting candidates (22.91 seconds)
Round 417: bisecting over non-failing examples 421-422 .. multiple culprits detected - splitting candidates (23.31 seconds)
Round 418: bisecting over non-failing examples 424-451 .. multiple culprits detected - splitting candidates (23.13 seconds)
Round 419: bisecting over non-failing examples 424-437 .. multiple culprits detected - splitting candidates (21.97 seconds)
Round 420: bisecting over non-failing examples 424-430 .. multiple culprits detected - splitting candidates (20.09 seconds)
Round 421: bisecting over non-failing examples 424-427 .. multiple culprits detected - splitting candidates (20.29 seconds)
Round 422: bisecting over non-failing examples 424-425 .. multiple culprits detected - splitting candidates (22.86 seconds)
Round 423: bisecting over non-failing examples 426-427 .. multiple culprits detected - splitting candidates (20.2 seconds)
Round 424: bisecting over non-failing examples 428-430 .. multiple culprits detected - splitting candidates (19.25 seconds)
Round 425: bisecting over non-failing examples 428-429 .. multiple culprits detected - splitting candidates (21.99 seconds)
Round 426: bisecting over non-failing examples 431-437 .. multiple culprits detected - splitting candidates (22.71 seconds)
Round 427: bisecting over non-failing examples 431-434 .. multiple culprits detected - splitting candidates (19.48 seconds)
Round 428: bisecting over non-failing examples 431-432 .. multiple culprits detected - splitting candidates (20.2 seconds)
Round 429: bisecting over non-failing examples 433-434 .. multiple culprits detected - splitting candidates (20.89 seconds)
Round 430: bisecting over non-failing examples 435-437 .. multiple culprits detected - splitting candidates (20.43 seconds)
Round 431: bisecting over non-failing examples 435-436 .. multiple culprits detected - splitting candidates (20.38 seconds)
Round 432: bisecting over non-failing examples 438-451 .. multiple culprits detected - splitting candidates (18.45 seconds)
Round 433: bisecting over non-failing examples 438-444 .. multiple culprits detected - splitting candidates (17.58 seconds)
Round 434: bisecting over non-failing examples 438-441 .. multiple culprits detected - splitting candidates (18.86 seconds)
Round 435: bisecting over non-failing examples 438-439 .. multiple culprits detected - splitting candidates (18.75 seconds)
Round 436: bisecting over non-failing examples 440-441 .. multiple culprits detected - splitting candidates (19.05 seconds)
Round 437: bisecting over non-failing examples 442-444 .. multiple culprits detected - splitting candidates (17.84 seconds)
Round 438: bisecting over non-failing examples 442-443 .. multiple culprits detected - splitting candidates (18.17 seconds)
Round 439: bisecting over non-failing examples 445-451 .. multiple culprits detected - splitting candidates (19.01 seconds)
Round 440: bisecting over non-failing examples 445-448 .. multiple culprits detected - splitting candidates (19.03 seconds)
Round 441: bisecting over non-failing examples 445-446 .. multiple culprits detected - splitting candidates (19.06 seconds)
Round 442: bisecting over non-failing examples 447-448 .. multiple culprits detected - splitting candidates (18.89 seconds)
Round 443: bisecting over non-failing examples 449-451 .. multiple culprits detected - splitting candidates (18.91 seconds)
Round 444: bisecting over non-failing examples 449-450 .. multiple culprits detected - splitting candidates (18.94 seconds)
Round 445: bisecting over non-failing examples 452-901 .. multiple culprits detected - splitting candidates (14.29 seconds)
Round 446: bisecting over non-failing examples 452-676 .. multiple culprits detected - splitting candidates (15.49 seconds)
Round 447: bisecting over non-failing examples 452-564 .. multiple culprits detected - splitting candidates (22.35 seconds)
Round 448: bisecting over non-failing examples 452-508 .. multiple culprits detected - splitting candidates (19.19 seconds)
Round 449: bisecting over non-failing examples 452-480 .. multiple culprits detected - splitting candidates (19.27 seconds)
Round 450: bisecting over non-failing examples 452-466 .. multiple culprits detected - splitting candidates (19.14 seconds)
Round 451: bisecting over non-failing examples 452-459 .. multiple culprits detected - splitting candidates (19.39 seconds)
Round 452: bisecting over non-failing examples 452-455 .. multiple culprits detected - splitting candidates (18.9 seconds)
Round 453: bisecting over non-failing examples 452-453 .. multiple culprits detected - splitting candidates (19.02 seconds)
Round 454: bisecting over non-failing examples 454-455 .. multiple culprits detected - splitting candidates (18.96 seconds)
Round 455: bisecting over non-failing examples 456-459 .. multiple culprits detected - splitting candidates (19.23 seconds)
Round 456: bisecting over non-failing examples 456-457 . ignoring example 456 (9.58 seconds)
Round 457: bisecting over non-failing examples 458-459 .. multiple culprits detected - splitting candidates (18.82 seconds)
Round 458: bisecting over non-failing examples 460-466 .. multiple culprits detected - splitting candidates (19 seconds)
Round 459: bisecting over non-failing examples 460-463 .. multiple culprits detected - splitting candidates (19.02 seconds)
Round 460: bisecting over non-failing examples 460-461 .. multiple culprits detected - splitting candidates (19.05 seconds)
Round 461: bisecting over non-failing examples 462-463 .. multiple culprits detected - splitting candidates (19.18 seconds)
Round 462: bisecting over non-failing examples 464-466 .. multiple culprits detected - splitting candidates (19.22 seconds)
Round 463: bisecting over non-failing examples 464-465 .. multiple culprits detected - splitting candidates (19.18 seconds)
Round 464: bisecting over non-failing examples 467-480 .. multiple culprits detected - splitting candidates (19.27 seconds)
Round 465: bisecting over non-failing examples 467-473 .. multiple culprits detected - splitting candidates (19.9 seconds)
Round 466: bisecting over non-failing examples 467-470 .. multiple culprits detected - splitting candidates (19.4 seconds)
Round 467: bisecting over non-failing examples 467-468 .. multiple culprits detected - splitting candidates (19.33 seconds)
Round 468: bisecting over non-failing examples 469-470 .. multiple culprits detected - splitting candidates (20.35 seconds)
Round 469: bisecting over non-failing examples 471-473 .. multiple culprits detected - splitting candidates (19.38 seconds)
Round 470: bisecting over non-failing examples 471-472 .. multiple culprits detected - splitting candidates (19.12 seconds)
Round 471: bisecting over non-failing examples 474-480 .. multiple culprits detected - splitting candidates (19.01 seconds)
Round 472: bisecting over non-failing examples 474-477 .. multiple culprits detected - splitting candidates (19.28 seconds)
Round 473: bisecting over non-failing examples 474-475 .. multiple culprits detected - splitting candidates (20.21 seconds)
Round 474: bisecting over non-failing examples 476-477 .. multiple culprits detected - splitting candidates (20.42 seconds)
Round 475: bisecting over non-failing examples 478-480 .. multiple culprits detected - splitting candidates (18.82 seconds)
Round 476: bisecting over non-failing examples 478-479 .. multiple culprits detected - splitting candidates (20.39 seconds)
Round 477: bisecting over non-failing examples 481-508 .. multiple culprits detected - splitting candidates (19.55 seconds)
Round 478: bisecting over non-failing examples 481-494 .. multiple culprits detected - splitting candidates (20.02 seconds)
Round 479: bisecting over non-failing examples 481-487 .. multiple culprits detected - splitting candidates (20.1 seconds)
Round 480: bisecting over non-failing examples 481-484 .. multiple culprits detected - splitting candidates (19.49 seconds)
Round 481: bisecting over non-failing examples 481-482 .. multiple culprits detected - splitting candidates (19.25 seconds)
Round 482: bisecting over non-failing examples 483-484 .. multiple culprits detected - splitting candidates (20.97 seconds)
Round 483: bisecting over non-failing examples 485-487 .. multiple culprits detected - splitting candidates (18.9 seconds)
Round 484: bisecting over non-failing examples 485-486 .. multiple culprits detected - splitting candidates (19.1 seconds)
Round 485: bisecting over non-failing examples 488-494 .. multiple culprits detected - splitting candidates (19.16 seconds)
Round 486: bisecting over non-failing examples 488-491 .. multiple culprits detected - splitting candidates (19.18 seconds)
Round 487: bisecting over non-failing examples 488-489 .. multiple culprits detected - splitting candidates (19.08 seconds)
Round 488: bisecting over non-failing examples 490-491 .. multiple culprits detected - splitting candidates (18.91 seconds)
Round 489: bisecting over non-failing examples 492-494 .. multiple culprits detected - splitting candidates (18.88 seconds)
Round 490: bisecting over non-failing examples 492-493 .. multiple culprits detected - splitting candidates (19.26 seconds)
Round 491: bisecting over non-failing examples 495-508 .. multiple culprits detected - splitting candidates (20.53 seconds)
Round 492: bisecting over non-failing examples 495-501 .. multiple culprits detected - splitting candidates (19.84 seconds)
Round 493: bisecting over non-failing examples 495-498 .. multiple culprits detected - splitting candidates (19.28 seconds)
Round 494: bisecting over non-failing examples 495-496 .. multiple culprits detected - splitting candidates (19.47 seconds)
Round 495: bisecting over non-failing examples 497-498 .. multiple culprits detected - splitting candidates (20 seconds)
Round 496: bisecting over non-failing examples 499-501 .. multiple culprits detected - splitting candidates (21.13 seconds)
Round 497: bisecting over non-failing examples 499-500 .. multiple culprits detected - splitting candidates (20.07 seconds)
Round 498: bisecting over non-failing examples 502-508 .. multiple culprits detected - splitting candidates (19.86 seconds)
Round 499: bisecting over non-failing examples 502-505 .. multiple culprits detected - splitting candidates (19.96 seconds)
Round 500: bisecting over non-failing examples 502-503 .. multiple culprits detected - splitting candidates (18.82 seconds)
Round 501: bisecting over non-failing examples 504-505 .. multiple culprits detected - splitting candidates (20.39 seconds)
Round 502: bisecting over non-failing examples 506-508 .. multiple culprits detected - splitting candidates (19.92 seconds)
Round 503: bisecting over non-failing examples 506-507 .. multiple culprits detected - splitting candidates (19.77 seconds)
Round 504: bisecting over non-failing examples 509-564 .. multiple culprits detected - splitting candidates (18.85 seconds)
Round 505: bisecting over non-failing examples 509-536 .. multiple culprits detected - splitting candidates (19.05 seconds)
Round 506: bisecting over non-failing examples 509-522 .. multiple culprits detected - splitting candidates (18.98 seconds)
Round 507: bisecting over non-failing examples 509-515 .. multiple culprits detected - splitting candidates (18.86 seconds)
Round 508: bisecting over non-failing examples 509-512 .. multiple culprits detected - splitting candidates (21.15 seconds)
Round 509: bisecting over non-failing examples 509-510 .. multiple culprits detected - splitting candidates (20.51 seconds)
Round 510: bisecting over non-failing examples 511-512 .. multiple culprits detected - splitting candidates (19.82 seconds)
Round 511: bisecting over non-failing examples 513-515 .. multiple culprits detected - splitting candidates (20.73 seconds)
Round 512: bisecting over non-failing examples 513-514 .. multiple culprits detected - splitting candidates (19.87 seconds)
Round 513: bisecting over non-failing examples 516-522 .. multiple culprits detected - splitting candidates (19.36 seconds)
Round 514: bisecting over non-failing examples 516-519 .. multiple culprits detected - splitting candidates (26.08 seconds)
Round 515: bisecting over non-failing examples 516-517 .. multiple culprits detected - splitting candidates (43.27 seconds)
Round 516: bisecting over non-failing examples 518-519 . ignoring example 518 (37.89 seconds)
Round 517: bisecting over non-failing examples 520-522 .. multiple culprits detected - splitting candidates (59.87 seconds)
Round 518: bisecting over non-failing examples 520-521 . ignoring example 520 (28.43 seconds)
Round 519: bisecting over non-failing examples 523-536 . ignoring examples 523-529 (37.83 seconds)
Round 520: bisecting over non-failing examples 530-536 . ignoring examples 530-533 (24.53 seconds)
Round 521: bisecting over non-failing examples 534-536 .. multiple culprits detected - splitting candidates (59.78 seconds)
Round 522: bisecting over non-failing examples 534-535 .. multiple culprits detected - splitting candidates (41.32 seconds)
Round 523: bisecting over non-failing examples 537-564 .. multiple culprits detected - splitting candidates (20.64 seconds)
Round 524: bisecting over non-failing examples 537-550 .. multiple culprits detected - splitting candidates (20.08 seconds)
Round 525: bisecting over non-failing examples 537-543 .. multiple culprits detected - splitting candidates (20.52 seconds)
Round 526: bisecting over non-failing examples 537-540 .. multiple culprits detected - splitting candidates (19.85 seconds)
Round 527: bisecting over non-failing examples 537-538 .. multiple culprits detected - splitting candidates (19.23 seconds)
Round 528: bisecting over non-failing examples 539-540 .. multiple culprits detected - splitting candidates (19.24 seconds)
Round 529: bisecting over non-failing examples 541-543 .. multiple culprits detected - splitting candidates (20.06 seconds)
Round 530: bisecting over non-failing examples 541-542 .. multiple culprits detected - splitting candidates (19.29 seconds)
Round 531: bisecting over non-failing examples 544-550 .. multiple culprits detected - splitting candidates (19.35 seconds)
Round 532: bisecting over non-failing examples 544-547 .. multiple culprits detected - splitting candidates (19.83 seconds)
Round 533: bisecting over non-failing examples 544-545 .. multiple culprits detected - splitting candidates (19.49 seconds)
Round 534: bisecting over non-failing examples 546-547 .. multiple culprits detected - splitting candidates (19.87 seconds)
Round 535: bisecting over non-failing examples 548-550 .. multiple culprits detected - splitting candidates (19.14 seconds)
Round 536: bisecting over non-failing examples 548-549 .. multiple culprits detected - splitting candidates (19.19 seconds)
Round 537: bisecting over non-failing examples 551-564 .. multiple culprits detected - splitting candidates (19.19 seconds)
Round 538: bisecting over non-failing examples 551-557 .. multiple culprits detected - splitting candidates (19.15 seconds)
Round 539: bisecting over non-failing examples 551-554 .. multiple culprits detected - splitting candidates (19.9 seconds)
Round 540: bisecting over non-failing examples 551-552 .. multiple culprits detected - splitting candidates (19.76 seconds)
Round 541: bisecting over non-failing examples 553-554 .. multiple culprits detected - splitting candidates (20.65 seconds)
Round 542: bisecting over non-failing examples 555-557 .. multiple culprits detected - splitting candidates (20.6 seconds)
Round 543: bisecting over non-failing examples 555-556 .. multiple culprits detected - splitting candidates (20.21 seconds)
Round 544: bisecting over non-failing examples 558-564 .. multiple culprits detected - splitting candidates (20.18 seconds)
Round 545: bisecting over non-failing examples 558-561 .. multiple culprits detected - splitting candidates (19.37 seconds)
Round 546: bisecting over non-failing examples 558-559 .. multiple culprits detected - splitting candidates (19.15 seconds)
Round 547: bisecting over non-failing examples 560-561 .. multiple culprits detected - splitting candidates (19.09 seconds)
Round 548: bisecting over non-failing examples 562-564 .. multiple culprits detected - splitting candidates (19.24 seconds)
Round 549: bisecting over non-failing examples 562-563 .. multiple culprits detected - splitting candidates (20.79 seconds)
Round 550: bisecting over non-failing examples 565-676 .. multiple culprits detected - splitting candidates (16.79 seconds)
Round 551: bisecting over non-failing examples 565-620 .. multiple culprits detected - splitting candidates (19.3 seconds)
Round 552: bisecting over non-failing examples 565-592 .. multiple culprits detected - splitting candidates (18.69 seconds)
Round 553: bisecting over non-failing examples 565-578 .. multiple culprits detected - splitting candidates (18.79 seconds)
Round 554: bisecting over non-failing examples 565-571 .. multiple culprits detected - splitting candidates (18.74 seconds)
Round 555: bisecting over non-failing examples 565-568 .. multiple culprits detected - splitting candidates (18.84 seconds)
Round 556: bisecting over non-failing examples 565-566 .. multiple culprits detected - splitting candidates (18.87 seconds)
Round 557: bisecting over non-failing examples 567-568 .. multiple culprits detected - splitting candidates (18.84 seconds)
Round 558: bisecting over non-failing examples 569-571 .. multiple culprits detected - splitting candidates (20.28 seconds)
Round 559: bisecting over non-failing examples 569-570 .. multiple culprits detected - splitting candidates (19.37 seconds)
Round 560: bisecting over non-failing examples 572-578 .. multiple culprits detected - splitting candidates (18.79 seconds)
Round 561: bisecting over non-failing examples 572-575 .. multiple culprits detected - splitting candidates (18.96 seconds)
Round 562: bisecting over non-failing examples 572-573 .. multiple culprits detected - splitting candidates (19.21 seconds)
Round 563: bisecting over non-failing examples 574-575 .. multiple culprits detected - splitting candidates (19.04 seconds)
Round 564: bisecting over non-failing examples 576-578 .. multiple culprits detected - splitting candidates (18.65 seconds)
Round 565: bisecting over non-failing examples 576-577 .. multiple culprits detected - splitting candidates (19.36 seconds)
Round 566: bisecting over non-failing examples 579-592 .. multiple culprits detected - splitting candidates (19.04 seconds)
Round 567: bisecting over non-failing examples 579-585 .. multiple culprits detected - splitting candidates (18.77 seconds)
Round 568: bisecting over non-failing examples 579-582 .. multiple culprits detected - splitting candidates (18.7 seconds)
Round 569: bisecting over non-failing examples 579-580 .. multiple culprits detected - splitting candidates (18.75 seconds)
Round 570: bisecting over non-failing examples 581-582 .. multiple culprits detected - splitting candidates (18.72 seconds)
Round 571: bisecting over non-failing examples 583-585 .. multiple culprits detected - splitting candidates (18.73 seconds)
Round 572: bisecting over non-failing examples 583-584 .. multiple culprits detected - splitting candidates (18.63 seconds)
Round 573: bisecting over non-failing examples 586-592 .. multiple culprits detected - splitting candidates (18.73 seconds)
Round 574: bisecting over non-failing examples 586-589 .. multiple culprits detected - splitting candidates (18.7 seconds)
Round 575: bisecting over non-failing examples 586-587 .. multiple culprits detected - splitting candidates (18.77 seconds)
Round 576: bisecting over non-failing examples 588-589 .. multiple culprits detected - splitting candidates (18.71 seconds)
Round 577: bisecting over non-failing examples 590-592 .. multiple culprits detected - splitting candidates (18.66 seconds)
Round 578: bisecting over non-failing examples 590-591 .. multiple culprits detected - splitting candidates (18.66 seconds)
Round 579: bisecting over non-failing examples 593-620 .. multiple culprits detected - splitting candidates (19.11 seconds)
Round 580: bisecting over non-failing examples 593-606 .. multiple culprits detected - splitting candidates (18.9 seconds)
Round 581: bisecting over non-failing examples 593-599 .. multiple culprits detected - splitting candidates (19.07 seconds)
Round 582: bisecting over non-failing examples 593-596 .. multiple culprits detected - splitting candidates (19.43 seconds)
Round 583: bisecting over non-failing examples 593-594 .. multiple culprits detected - splitting candidates (18.57 seconds)
Round 584: bisecting over non-failing examples 595-596 .. multiple culprits detected - splitting candidates (18.98 seconds)
Round 585: bisecting over non-failing examples 597-599 .. multiple culprits detected - splitting candidates (18.7 seconds)
Round 586: bisecting over non-failing examples 597-598 .. multiple culprits detected - splitting candidates (19.12 seconds)
Round 587: bisecting over non-failing examples 600-606 .. multiple culprits detected - splitting candidates (18.88 seconds)
Round 588: bisecting over non-failing examples 600-603 .. multiple culprits detected - splitting candidates (19.23 seconds)
Round 589: bisecting over non-failing examples 600-601 .. multiple culprits detected - splitting candidates (19.18 seconds)
Round 590: bisecting over non-failing examples 602-603 .. multiple culprits detected - splitting candidates (19.85 seconds)
Round 591: bisecting over non-failing examples 604-606 .. multiple culprits detected - splitting candidates (19.23 seconds)
Round 592: bisecting over non-failing examples 604-605 .. multiple culprits detected - splitting candidates (19.26 seconds)
Round 593: bisecting over non-failing examples 607-620 .. multiple culprits detected - splitting candidates (19.05 seconds)
Round 594: bisecting over non-failing examples 607-613 .

@myronmarston
Copy link
Member Author

OK, I figured out the source of the problem.

exit simply raises a SystemExit error. That means it unwinds the call stack and any ensure blocks are given a chance to run. That includes this one in the reporter. Naturally, the "Finished.." lines gets printed in both processes.

I'm not sure on what the right fix for this is. We could potentially treat SystemExit as special and allow the "Finish..." line to print in that case. Or we could store Process.pid when the spec suite starts and put logic in the reporter to not emit output if the current Process.pid is different (as that would imply that it's in a subprocess). However, we'd have to be very careful how we do that. Pre-forking runners like spring have historically had a hard time properly resetting initial state such as the rspec start time, and if we did it wrong, it might break RSpec in such situations. Or we could decide that fork and exit work how they work and we shouldn't work around them.

Any thoughts, @rspec/rspec? @SephVelut, as the one who opened #2058, what do you think?

@SephVelut
Copy link

Good dig into the problem @myronmarston!

Testing a process that happens to fork and exit on the same tty - with a stack that contains callbacks to rspec - is definitely a gotcha. You'll wonder why you're getting multiple runs of the tests. But I think its probably an unavoidable gotcha when testing forks. Tracking the pid on the reporter is an interesting idea. But I can also se why Process.pid would be a precarious moving part. It would come down to asking @rspec/rspec if the statement "rspec will never emit output in a process different from the one that initiated rspec" is okay.

Or we could decide that fork and exit work how they work and we shouldn't work around them.

I'm inclined to agree. I suppose even a feature for 'turning off' reporting in subprocess that exits won't be a better solution than simply using begin rescue.

Here is my current suggestion. Lets add a note to gotchas and suggest manually rescuing SystemExit with exit!. We should include the keywords `multiple runs/output' etc. A 2-second Google search when I first encountered this could have saved me a lot of time.

@xaviershay
Copy link
Member

  1. Why do we even need that ensure block in reporter? If I ctrl+c, OOM, or w/e I wouldn't expect them to be run. And if a spec raises an exception it should be caught earlier, right?
  2. "Or we could decide that fork and exit work how they work and we shouldn't work around them." This is my default position unless convinced otherwise.
  3. ""rspec will never emit output in a process different from the one that initiated rspec" is okay." that does seem reasonable, though I think point 2 wins out for me.
  4. For this problem, I think we should recommend people test using a subprocess. This is what we do in our suite for example with testing monkey-patches, CLIs, and the like.

@myronmarston
Copy link
Member Author

Why do we even need that ensure block in reporter? If I ctrl+c, OOM, or w/e I wouldn't expect them to be run. And if a spec raises an exception it should be caught earlier, right?

I'm not sure if we need it for when users are running specs via the rspec command. However, that method is a public API and users may use an alternate runner that uses the APIs provided by rspec in a different way. The ensure is there so that the start and finish are paired even if an exception occurs.

I tried removing the ensure locally and it didn't break anything so it might be safe to remove...but the fact that it's a public API means it could affect others.

Removing the ensure against this PR branch fixes the problem I was having, so it's something I want to consider more.

"Or we could decide that fork and exit work how they work and we shouldn't work around them." This is my default position unless convinced otherwise.

Mine, too. (Particularly since I've not heard of anyone else running into this gotcha).

Here is my current suggestion. Lets add a note to gotchas and suggest manually rescuing SystemExit with exit!. We should include the keywords `multiple runs/output' etc. A 2-second Google search when I first encountered this could have saved me a lot of time.

What gotchas list do you want a note added to? You linked to capybara.

Anyhow, what you suggested made me realize this would work:

RSpec.configure do |c|
  c.around(:example) do |ex|
    begin
      ex.run
    rescue SystemExit
      exit!
    end
  end
end

You could use a hook like this to transform an exit into an exit!. I tried this for the example spec I added in this PR and it worked.

As for this PR...regardless of if we do anything in particular for the SystemExit case, I think the change here is sound. While we do not want to rescue only StandardError-subclassed exceptions, there are some Exception subclasses we should not be rescueing and the change here is a good way to stop doing that. As a point of comparison, you can see what Minitest does (it's where I got the list):

@myronmarston
Copy link
Member Author

I looked a bit more into the ensure we've been discussing, and it looks like it was added to make ctrl-c work properly -- see here.

Ctrl-C is one of those things that's tricky to test in a realistic manner so the fact that removing the ensure does not break any tests doesn't mean it's not needed. It would take a bunch of effort to figure out if it's still needed for that case and I'd rather spend my limited open source time on other things.

@xaviershay
Copy link
Member

fwiw I always wanted ctrl+c to just quit immediately, I never found the do-it-twice requirement helpful :/

As for this PR...regardless of if we do anything in particular for the SystemExit case, I think the change here is sound.

Agree

I'd rather spend my limited open source time on other things.

Legit.

@myronmarston
Copy link
Member Author

fwiw I always wanted ctrl+c to just quit immediately, I never found the do-it-twice requirement helpful :/

It's not required to do it twice (or at least it shouldn't be). When you hit ctrl-c RSpec immediately begins aborting but tries to run all the cleanup logic (e.g. ensure blocks all the way up the stack, after hooks, etc) and then print the summary line (showing how many tests passed and failed). It treats a 2nd ctrl-c as a signal to not bother with the cleanup or summary.

In my experience, the 1st ctrl-c does the cleanup exit pretty quickly as long as it's not a large project with lots of cleanup to perform. I just tried it while running bin/rspec against rspec-mocks and it felt around a couple hundred ms. A bunch of that is actually simplecov's at_exit hook producing a coverage report. Running NO_COVERAGE=1 bin/rspec and then hitting ctrl-c makes it exit considerably faster.

Your projects are perhaps just quite large? Or maybe you've hit a genuine bug no one's reported where RSpec isn't able to exit on the first ctrl-c for some reason.

@xaviershay
Copy link
Member

it's the large projects thing. Actually when I think about it the current behaviour is a good default for most people.

@SephVelut
Copy link

What gotchas list do you want a note added to? You linked to capybara.

Yeah, my bad. In any case this thread's existence is probably good enough as I'd like to think most people would eventually conduct a search on the issue page.

@@ -227,14 +227,14 @@ def run(example_group_instance, reporter)
rescue Pending::SkipDeclaredInExample
# no-op, required metadata has already been set by the `skip`
# method.
rescue Exception => e
rescue Support::AllExceptionsExceptOnesWeMustNotRescue => e
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why, but somehow this change causes a failure for this cucumber scenario on 1.8.7:

https://travis-ci.org/rspec/rspec-core/jobs/77979590

If I change this back to Exception that passes. Makes absolutely no sense to me :(.

Anyone got any ideas why?

@myronmarston myronmarston force-pushed the stop-rescuing-all-exceptions branch 3 times, most recently from 8cfd75e to bfea12d Compare October 7, 2015 16:12
We do want to generally rescue `Exception` and
any subclasses but there are a few exceptions
we should not rescue. For example, if we rescue
`SystemExit`, it prevents `exit` from exiting.

Part of #2058.
@myronmarston
Copy link
Member Author

This is finally green! Anyone from @rspec/rspec want to do a final review before I merge?

set_exception(e)
ensure
run_after_example
end
end
end
rescue Exception => e
rescue Support::AllExceptionsExceptOnesWeMustNotRescue => e
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Namespaced here but not above?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The one above is RSpec::Core::AllExceptionsExcludingDangerousOnesOnRubiesThatAllowIt, not a constant from the RSpec::Support namespace. See the definition and comment below. Yes, it's weird.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only this one? Wut!?

@xaviershay
Copy link
Member

LGTM

myronmarston added a commit that referenced this pull request Oct 7, 2015
@myronmarston myronmarston merged commit 889d35f into master Oct 7, 2015
@myronmarston myronmarston deleted the stop-rescuing-all-exceptions branch October 7, 2015 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants