Non-interactive commands on remote host fail

[danepowell]

Non-interactive Drush commands run against a remote site alias will fail with a fairly obscure error.

In my example, @blted91.local is a site inside of a DrupalVM:

local:
    root: /var/www/blted91/docroot
    uri: 'http://local.blted91.com'
    host: local.blted91.com
    user: vagrant
    ssh:
      options: '-o PasswordAuthentication=no -i /home/dane/.vagrant.d/insecure_private_key'

Running drush @blted91.local status --no-interaction produces this:

Drupal version   : 8.4.0                                                 
 Site URI         : http://local.blted91.com                              
 DB driver        : mysql                                                 
 DB hostname      : localhost                                             
 DB port          : 3306                                                  
 DB username      : drupal                                                
 DB name          : drupal                                                
 Database         : Connected                                             
 Drupal bootstrap : Successful                                            
 Default theme    : bartik                                                
 Admin theme      : seven                                                 
 PHP binary       : /usr/bin/php7.1                                       
 PHP config       : /etc/php/7.1/cli/php.ini                              
 PHP OS           : Linux                                                 
 Drush script     : /var/www/blted91/vendor/drush/drush/drush             
 Drush version    : 9.0.0-beta8                                           
 Drush temp       : /tmp                                                  
 Drush configs    : /var/www/blted91/vendor/drush/drush/drush.yml         
                    /var/www/blted91/drush/drush.yml                      
 Drush aliases    : /var/www/blted91/drush/site-aliases/blted91.alias.yml 
 Install profile  : lightning                                             
 Drupal root      : /var/www/blted91/docroot                              
 Site path        : sites/default                                         
 Files, Public    : sites/default/files                                   
 Files, Private   : /var/www/blted91/files-private                        
 Files, Temp      : /tmp                                                  
 [error]  The command could not be executed successfully (returned:  Drupal version   : 8.4.0                                                 
 Site URI         : http://local.blted91.com                              
 DB driver        : mysql                                                 
 DB hostname      : localhost                                             
 DB port          : 3306                                                  
 DB username      : drupal                                                
 DB name          : drupal                                                
 Database         : Connected                                             
 Drupal bootstrap : Successful                                            
 Default theme    : bartik                                                
 Admin theme      : seven                                                 
 PHP binary       : /usr/bin/php7.1                                       
 PHP config       : /etc/php/7.1/cli/php.ini                              
 PHP OS           : Linux                                                 
 Drush script     : /var/www/blted91/vendor/drush/drush/drush             
 Drush version    : 9.0.0-beta8                                           
 Drush temp       : /tmp                                                  
 Drush configs    : /var/www/blted91/vendor/drush/drush/drush.yml         
                    /var/www/blted91/drush/drush.yml                      
 Drush aliases    : /var/www/blted91/drush/site-aliases/blted91.alias.yml 
 Install profile  : lightning                                             
 Drupal root      : /var/www/blted91/docroot                              
 Site path        : sites/default                                         
 Files, Public    : sites/default/files                                   
 Files, Private   : /var/www/blted91/files-private                        
 Files, Temp      : /tmp                                                  
, code: 0)

In my debugging, I think I narrowed this down to a problem with drush_backend_parse_output()... at least that's where the process falls apart. It's also possible that it's working fine and something else is not injecting the correct separator tokens for it to parse.

[greg-1-anderson]

Are the parameters used in the ssh call the same in working and non-working cases? Run with --debug to see the whole ssh call.

[danepowell]

working:

drush @blted91.local status --debug

ssh -o PasswordAuthentication=no -i /home/dane/.vagrant.d/insecure_private_key -t vagrant@local.blted91.com 'drush --verbose --debug --root=/var/www/blted91/docroot --uri=http://local.blted91.com status --debug 2>&1' 2>&1

not working:

drush @blted91.local status --no-interaction --debug

ssh -o PasswordAuthentication=no -i /home/dane/.vagrant.d/insecure_private_key vagrant@local.blted91.com 'drush --verbose --debug --root=/var/www/blted91/docroot --uri=http://local.blted91.com status --no-interaction --debug 2>&1' 2>&1

[weitzman]

I can confirm the bug. Need some input from @greg-1-anderson in order to fix it.

In the failing case (the non-interactive one), it looks to me like the original request is expecting the remote to send its output with backend delimiters and that whole ceremony. But it never tells the remote to do that as it does not send the --backend option. Thats evident in the failing command thats above.

I'm curious about this line -

drush/src/Runtime/RedispatchHook.php

Line 96 in fac6079

'backend' => false,

. Should that be set to true under some circumstances? Or maybe backend.inc should be doing that and isn't?

[weitzman]

Posted a possible fix to #3136. The test fails there might be hard to fix without breaking other stuff

[greg-1-anderson]

So, the fact that Drush is not sending --backend in this instance is correct. A non-interactive call via the commandline is still not a backend call. I think that the problem here is that backend_parse_output is being called, and it should not be, because this is not a backend call.

I'll have to compare execution on Drush 8 to be sure.

[greg-1-anderson]

I think the problem here is that Drush's nomenclature is confusing around the issue of "interactive" vs. "incremental output" vs "backend output".

Backend mode:

• Log messages are sent via JSON packets and printed incrementally
• Command output is sent as-is, printed incrementally AND accumulated
• The final command result is sent as a JSON packet

Interactive mode:

• No json packets are sent
• Log messages and command output are sent as-is and printed normally
• A tty is created so that interactive programs like vi will work

Incremental output:

• Exactly like interactive mode in most respects
• No tty is created
• --no-interaction is sent to the remote end. (Maybe this should be converted to -n in case remote end is Drush 8)

It seems like the problem here is that the remote end is correctly in incremental output mode, but the client gets confused and thinks this is a backend call. I need to isolate further, though.

[rbayliss]

This also happens when you invoke a remote command from a non-tty environment (eg: Invoking a drush command during NodeJS's child_process.spawn). Makes sense, since the interaction is something Symfony Console determines automatically based on whether it's speaking to a TTY.

[rbayliss]

I should also note that I can't seem to find any workaround for this for making the output of remote commands machine-readable in a non-TTY environment. Even if I were to somehow convince the application to set interactive = TRUE, SSH adds some garbage to stdout when it believes it's speaking to a TTY:

drush @prod sql:conf --format=json

# Output will include "Connection to XXX closed" at the end

drush @prod sql:conf --format=json --ssh-options="-o LogLevel=QUIET"

# Output starts with ^@^@ (characters are added by SSH due to believing it's speaking to a TTY)

drush @prod sql:conf --format=json --ssh-options="-o LogLevel=QUIET" --no-interaction

# Fails with "The command could not be executed successfully", as described above.

The extra stuff added by SSH is definitely outside the scope of this issue, but I'm just pointing out that there's currently no way to get machine readable output back from a remote command.

[greg-1-anderson]

I compared the data being passed to drush_backend_invoke_concurrent in Drush 8 and Drush 9, and from there came up with #3407, which hopefully will be sufficient to resolve these issues.

[weitzman]

This problem is being bypassed with new site-process API in #3758.