TASK [template_service_broker : Reconcile with RBAC file] fails

[adelton]

Description

Provide a brief description of your issue here.

On a RHEL 7.4 machine, running

ansible-playbook -i ansible-inventory /mnt/tests/OpenShift/Origin/install/dist/openshift-ansible/playbooks/byo/config.yml

fails.

Version

Please put the following version information in the code block
indicated below.

• Your ansible version per ansible --version

ansible 2.4.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, May  3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)]

If you're operating from a git clone:

• The output of git describe

openshift-ansible-3.7.5-1-4-g34dfb0d

If you're running from playbooks installed via RPM or
atomic-openshift-utils

• The output of rpm -q atomic-openshift-utils openshift-ansible

Place the output between the code block below:

N/A

Steps To Reproduce

1. ansible-playbook -i ansible-inventory /mnt/tests/OpenShift/Origin/install/dist/openshift-ansible/playbooks/byo/config.yml

Expected Results

Describe what you expected to happen.

No error, ansible-playbook passing and all-on-one OpenShift setup created.

Observed Results

Describe what is actually happening.

TASK [template_service_broker : Set default image variables based on deployment type] ***
ok: [master.example.com] => (item=/mnt/tests/OpenShift/Origin/install/dist/openshift-ansible/roles/template_service_broker/vars/default_images.yml)

TASK [template_service_broker : set template_service_broker facts] *************
ok: [master.example.com]

TASK [template_service_broker : oc_project] ************************************
changed: [master.example.com]

TASK [template_service_broker : command] ***************************************
ok: [master.example.com]

TASK [template_service_broker : copy] ******************************************
changed: [master.example.com] => (item=apiserver-template.yaml)
changed: [master.example.com] => (item=rbac-template.yaml)
changed: [master.example.com] => (item=template-service-broker-registration.yaml)
changed: [master.example.com] => (item=apiserver-config.yaml)

TASK [template_service_broker : yedit] *****************************************
ok: [master.example.com]

TASK [template_service_broker : slurp] *****************************************
ok: [master.example.com]

TASK [template_service_broker : Apply template file] ***************************
changed: [master.example.com]

TASK [template_service_broker : Reconcile with RBAC file] **********************
fatal: [master.example.com]: FAILED! => {"changed": true, "cmd": "oc process -f \"/tmp/tsb-ansible-KmuHhc/rbac-template.yaml\" | oc auth reconcile -f -", "delta": "0:00:00.700965", "end": "2017-11-09 22:50:01.727255", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2017-11-09 22:50:01.026290", "stderr": "Error: unknown shorthand flag: 'f' in -f\n\n\nUsage:\n  oc auth [options]\n\nAvailable Commands:\n  can-i       Check whether an action is allowed\n\nUse \"oc <command> --help\" for more information about a given command.\nUse \"oc options\" for a list of global command-line options (applies to all commands).", "stderr_lines": ["Error: unknown shorthand flag: 'f' in -f", "", "", "Usage:", "  oc auth [options]", "", "Available Commands:", "  can-i       Check whether an action is allowed", "", "Use \"oc <command> --help\" for more information about a given command.", "Use \"oc options\" for a list of global command-line options (applies to all commands)."], "stdout": "", "stdout_lines": []}
	to retry, use: --limit @/mnt/tests/OpenShift/Origin/install/dist/openshift-ansible/playbooks/byo/config.retry

PLAY RECAP *********************************************************************
master.example.com : ok=624  changed=263  unreachable=0    failed=1   
localhost                  : ok=12   changed=0    unreachable=0    failed=0   


INSTALLER STATUS ***************************************************************
Initialization             : Complete
Health Check               : Complete
etcd Install               : Complete
Master Install             : Complete
Master Additional Install  : Complete
Node Install               : Complete
Hosted Install             : Complete
Service Catalog Install    : In Progress
	This phase can be restarted by running: playbooks/byo/openshift-cluster/service-catalog.yml



Failure summary:


  1. Hosts:    master.example.com
     Play:     Service Catalog
     Task:     Reconcile with RBAC file
     Message:  non-zero return code

For long output or logs, consider using a gist

Additional Information

Provide any additional information which may help us diagnose the
issue.

• Your operating system and version, ie: RHEL 7.2, Fedora 23 ($ cat /etc/redhat-release)

Red Hat Enterprise Linux Server release 7.4 (Maipo)

• Your inventory file (especially any non-standard configuration parameters)

[OSEv3:children]
masters
nodes

[OSEv3:vars]
ansible_ssh_user=root
deployment_type=origin
openshift_disable_check=memory_availability,docker_storage,disk_availability,package_version

osm_cluster_network_cidr=10.128.0.0/14
openshift_portal_net=172.30.0.0/16
osm_host_subnet_length=9

openshift_router_selector='node-router=true'
openshift_registry_selector='node-registry=true'

openshift_master_identity_providers=[{'name': 'allow_all', 'login': 'true', 'challenge': 'true', 'kind': 'AllowAllPasswordIdentityProvider'}]

[masters]
master.example.com

[etcd]
master.example.com

[nodes]
master.example.com openshift_schedulable=true openshift_node_labels="{ 'node-router': 'true', 'node-registry': 'true', 'node-app': 'true' }"

• Sample code, etc

The failure started to show up on 2017-11-09 morning and is determinstic. Previous runs up until 2017-11-07 evening were passing.

[adelton]

In those passing jobs (until 2017-11-07) I see

TASK [template_service_broker : Reconcile with RBAC file] **********************
skipping: [master.example.com]

which might be the reason why we did not see the failure earlier.

[adelton]

I have got

# rpm -qf /usr/bin/oc
origin-clients-3.6.1-1.0.008f2d5.x86_64

and oc auth reconcile -f seems to be a 3.7 thing: openshift/origin#15412, https://github.com/openshift/openshift-docs/pull/5725/files.

But that origin-clients-3.6.1 was installed by the ansible playbook, so it should know it has installed old version ...

[ccollicutt]

I have this issue as well...any thoughts on how I can get past this? :)

[tomassedovic]

I haven't tested/verified this yet but this commit seems to have introduced the breakage:

ad977b7

[tomassedovic]

Yep, it's commit ad977b7.

I've tried the one before that, i.e. commit 56b529e and the ansible run succeeded, while commit ad977b7 failed.

[adelton]

@tomassedovic, I don't think that's the one. Semantially, it's still oc auth reconcile -f -, before the commit and after that, which is bound to fail on 3.6.

That commit is part of some bigger branch which AFAICT causes TASK [template_service_broker : Reconcile with RBAC file] to be no longer skipped, like it was before.

[helletheone]

i can confirm the problem, i have this problem too

[nikolicjakov]

I can also confirm this issue, just cloned the master branch and got this failure.

Fedora 27.1 Atomic Host

[adelton]

@simo5, we seem to be hitting oc auth reconcile code path with 3.6 OpenShift Origin that origin-ansible is using. Do you have any insight what the proper solution might be? This is currently blocking my ansible-based tests.

[rberlind]

Why does the release-3.7 branch even start by deploying 3.6 and then eventually upgrading to 3.7? It would seem better to start with 3.7.

[stevekuznetsov]

We do both. The vast majority of clusters will be upgraded from the last release, so just doing a fresh install of the newest code simply doesn't give us the signal we need.

[rberlind]

Thanks, @stevekuznetsov Is there an option I can set that would start a new cluster on 3.7? That might avoid the error discussed in this issue for some of us.

[stevekuznetsov]

I would not expect a brand new install to require a reconciliation. /cc @sdodson

[rberlind]

It might help if I explain that I was using openshift-ansible indirectly as part of https://github.com/dwmkerr/terraform-aws-openshift which first uses Terraform to provision some AWS resources and then uses openshift-ansible. The key interaction with openshift-ansible is in the install-from-bastion.sh script which invokes ANSIBLE_HOST_KEY_CHECKING=False /usr/local/bin/ansible-playbook -i ./inventory.cfg ./openshift-ansible/playbooks/byo/config.yml. The version in GitHub uses branch release-3.6, and that runs fine. But when I modified the script to specify branch release-3.7, I encountered the error described in this issue. And running oc version showed that OpenShift 3.6.x and Kubernetes 1.6.x had been installed.

[sdodson]

Do you have the ability to specify any openshift-ansible variables? Setting openshift_repos_enable_testing=true will enable the CentOS PaaS SIG testing repos which are currently the only source for 3.7 builds. It may be several days before they get 3.7.0 built and tested. Right now it's got 3.7.0 rc.0

[rberlind]

I could probably do that. But isn't the release-3.7 branch of openshift-ansible pointing to some version of 3.7? even if rc? My impression from the openshift-ansible README.md had been that using branch release-3.7 would use OpenShift 3.7 and therefore Kubernetes 1.7.x.

I'm not doing any of this for production. Just for being able to do a demo on OpenShift that uses HashiCorp's Terraform to provision the infrastructure and Vault to secure secrets. (I work at HashiCorp as an SE.) Using 3.7 would have been better for me since Kubernetes 1.7 defaults to using --service-account-lookup on the API Server. But I was able to enable that with OpenShift 3.6 in this setup today. So, using 3.7 is no longer so pressing.

I do appreciate the suggestions and help.

[leseb]

Same issue here on 6b6b422.

[adelton]

I confirm that using openshift_repos_enable_testing=true in the [OSEv3:vars] section of the inventory file makes things pass, and install

origin-3.7.0-0.rc.0.0.e92d5c5.x86_64
origin-clients-3.7.0-0.rc.0.0.e92d5c5.x86_64
origin-docker-excluder-3.7.0-0.rc.0.0.e92d5c5.noarch
origin-excluder-3.7.0-0.rc.0.0.e92d5c5.noarch
origin-master-3.7.0-0.rc.0.0.e92d5c5.x86_64
origin-node-3.7.0-0.rc.0.0.e92d5c5.x86_64
origin-sdn-ovs-3.7.0-0.rc.0.0.e92d5c5.x86_64
tuned-profiles-origin-node-3.7.0-0.rc.0.0.e92d5c5.x86_64

on my RHEL 7.4.

Is it expected that master branch no longer supports 3.6 and that openshift_repos_enable_testing=true has to be used? What do all the pull request tests use, that same openshift_repos_enable_testing=true setup?

[leseb]

@adelton thanks, this is fixing my deployment too.

[rberlind]

Thanks for the solution @sdodson and for testing @adelton

[sdodson]

@adelton The CI tests build origin from source on every pull request.
@rberlind one would think that, but unfortunately right now it's really controlled by the repos that are enabled, i think we can fix this though. We'd previously never insisted on knowing which version you intend to install but we'll soon start requiring that the release is set so we can use that instead. We can also set a default per branch so that each branch installs the desired version.

[adelton]

@adelton The CI tests build origin from source on every pull request.

Ah, so they do not use the repos? Wouldn't it be nice, for the openshift-ansible repo PR/CI tests, to (also) check that it works with the existing Origin repo bits? After all, that's what users will see and use.

[rberlind]

I just tested with git clone -b release-3.7 https://github.com/rberlind/openshift-ansible and openshift_repos_enable_testing=true in my inventory template cfg file. I did not get the "Reconcile with RBAC file" error that I had previously seen and which this ticket originally reported. However, the next task "Verify that TSB is running" failed after 120 attempts.

The specific error was: (7) Failed connect to apiserver.openshift-template-service-broker.svc:443; Connection refused"

It seems to me there are 2 problems here:

1. The DNS apiserver.openshift-template-service-broker.svc is not being resolved.
2. Should port 8443 be used?

If I add the DNS to /etc/hosts on master, I can do following:
$curl -k https://apiserver.openshift-template-service-broker.svc:8443/healthz
ok

But maybe that is hitting something other than the template-service-broker after I

Note that ps -ef shows /usr/bin/openshift start template-service-broker --secure-port=8443

Also, when I run journalctl -u origin-master-api.service, I see many entries like the following, suggesting an RBAC issue.

Dec 01 13:05:22 ip-10-0-1-219.ec2.internal atomic-openshift-master-api[11848]: I1201 13:05:22.293422 11848 rbac.go:116] RBAC DENY: user "system:serviceaccount:openshift-template-service-broker:apiserver" groups ["system:serviceaccounts" "system:serviceaccoun ...

Any ideas?

[rberlind]

One other possibility here is that my inventory.cfg file does not have openshift_template_service_broker_namespaces set.

https://docs.openshift.org/latest/install_config/install/advanced_install.html#configuring-template-service-broker suggests that it must be set:

"To configure the TSB, one or more projects must be defined as the broker’s source namespace(s) for loading templates and image streams into the service catalog. Set the desired projects by modifying the following in your inventory file’s [OSEv3:vars] section:
openshift_template_service_broker_namespaces=['openshift','myproject']

[rberlind]

That did not help, but I was able to install OpenShift Origin 3.7 by disabling the service catalog and TSB with:

openshift_enable_service_catalog=false
template_service_broker_install=false

[adelton]

Unfortunately, on Fedora 27, the openshift_repos_enable_testing=true does not help -- I still see

# rpm -qa '*origin*' | sort
origin-3.6.0-1.fc27.x86_64
origin-clients-3.6.0-1.fc27.x86_64
origin-docker-excluder-3.6.0-1.fc27.noarch
origin-excluder-3.6.0-1.fc27.noarch
origin-master-3.6.0-1.fc27.x86_64
origin-node-3.6.0-1.fc27.x86_64
origin-sdn-ovs-3.6.0-1.fc27.x86_64
tuned-profiles-origin-node-3.6.0-1.fc27.x86_64

installed and I still get the same oc auth reconcile -f - error.

[plaimbock]

@adelton koji only seems to have origin-3.6.0 for F27 so that would explain why you are getting 3.6.0. See https://koji.fedoraproject.org/koji/packageinfo?packageID=21564

[WoLfulus]

Anyone managed to get this working? We just started evaluating OpenShift and this is totally breaking our experience. We thought we were doing something wrong at first :(

[rberlind]

As I mentioned, I was able to get OpenShift 3.7 installed and running, but I had to set the following flags to disable the service catalog and template service

openshift_enable_service_catalog=false
template_service_broker_install=false

In that sense, I did not get OpenShift 3.7 fully deployed.

[bogdando]

openshift_repos_enable_testing: true didn't help for Centos 7.3.1611 case as well

[openshift@master-0 ~]$ rpm -qa '*origin*' | sort
origin-3.6.1-1.0.008f2d5.x86_64
origin-clients-3.6.1-1.0.008f2d5.x86_64
origin-docker-excluder-3.6.1-1.0.008f2d5.noarch
origin-excluder-3.6.1-1.0.008f2d5.noarch
origin-master-3.6.1-1.0.008f2d5.x86_64
origin-node-3.6.1-1.0.008f2d5.x86_64
origin-sdn-ovs-3.6.1-1.0.008f2d5.x86_64
tuned-profiles-origin-node-3.6.1-1.0.008f2d5.x86_64

[michaelgugino]

It appears CentOS has not pusblished a 3.7.0 RPM for openshift yet, even in the testing repo (I only see alpha and rc0). We will have to investigate the work-around with the older client as @adelton noted, we want 3.6 to be deployable on 3.7.

@WoLfulus in the mean time, if you a just trying to test out openshift and openshift-ansible, 3.6 is already available, I recommend starting with that branch for now until 3.7 RPMs are published.

[bogdando]

@michaelgugino there is openshift-clients listed in https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin/
yet it can't be picked up by yum update. I'm not a packaging guy to say what is wrong here though.

[michaelgugino]

@bogdando When I view that page, I only see an alpha and rc release for that package. The package version derived from the release, I'm not sure it will pick up alpha and rc packages by default.

[bogdando]

@michaelgugino with centos 7.4 (and openshift_repos_enable_testing: true), yum provides oc shows the client 3.7, so I hope this is what I wanted.

/update: Yeah, e2e worked for me

[sdodson]

@deads2k Can you pinpoint which release would resolve this error? It seems that oc auth may not have always had the -f flag?

TASK [template_service_broker : Reconcile with RBAC file] **********************
fatal: [master.example.com]: FAILED! => {"changed": true, "cmd": "oc process -f \"/tmp/tsb-ansible-KmuHhc/rbac-template.yaml\" |
oc auth reconcile -f -", "delta": "0:00:00.700965", "end": "2017-11-09 22:50:01.727255", "failed": true, "msg": "non-zero return code",
 "rc": 1, "start": "2017-11-09 22:50:01.026290", "stderr": "Error: unknown shorthand flag: 'f' in -f\n\n\nUsage:\n  oc auth [options]\n\n
Available Commands:\n  can-i       Check whether an action is allowed\n\nUse \"oc <command> --help\" for more information about
a given command.\nUse \"oc options\" for a list of global command-line options (applies to all commands).", "stderr_lines": ["Error:
unknown shorthand flag: 'f' in -f", "", "", "Usage:", "  oc auth [options]", "", "Available Commands:", "  can-i       Check whether an action
is allowed", "", "Use \"oc <command> --help\" for more information about a given command.", "Use \"oc options\" for a list of global
command-line options (applies to all commands)."], "stdout": "", "stdout_lines": []}
	to retry, use: --limit @/mnt/tests/OpenShift/Origin/install/dist/openshift-ansible/playbooks/byo/config.retry

The task that it's running is the following

# reconcile with rbac
- name: Reconcile with RBAC file
  shell: >
    {{ openshift.common.client_binary }} process -f "{{ mktemp.stdout }}/{{ __tsb_rbac_file }}" | {{ openshift.common.client_binary }} auth reconcile -f -

[adelton]

vendor/k8s.io/kubernetes/pkg/kubectl/cmd/auth/reconcile.go is new in OpenShift 3.7.

[tracestats]

I try to install the OpenShift 3.6 too and i have this issue as well, can anybody help to solve this issue?

[michaelgugino]

@tracestats use the release-3.6 branch. We recommend using the release branch that corresponds to the release of openshift you desire to deploy.

[tracestats]

@michaelgugino thank you! It works :)

[plaimbock]

Using branch release-3.6 to deploy oo36 worked. Thanks for the tip!

[sumitshatwara]

I also faced same issue around service catalog. I disabled below parameters as mentioned by @rberlind in host inventory file:

openshift_enable_service_catalog=false
template_service_broker_install=false

And the deployment of OpenShift Origin 3.7 was successful:

INSTALLER STATUS *******************************************************************************************************************************************************
Initialization : Complete
Health Check : Complete
etcd Install : Complete
Master Install : Complete
Master Additional Install : Complete
Node Install : Complete
Hosted Install : Complete

My question: Is service catalog a mandatory feature for OpenShift environment?
Use Case: I want to test FlexVolume driver of K8s only.

[adelton]

I no longer test 3.7 installs with openshift-ansible, closing.