Sunday, May 11, 2014

OpenStack Havana - Recovering Services Account When Deleted

I was working with one of my colleagues who had accidentally deleted the services account with OpenStack.   Unfortunately if this happens, it tends to break your setup in a really big way.   After opening a case with Redhat whose OpenStack distribution we were using led to no results.  I managed to reverse engineer where the services account resided and reestablish it.  Here are the steps I did:

Symptoms:

1) In web gui user gets "Oops something went wrong!" when trying to login.   User can get get valid token at command line (keystone token-get) but authorization fails.
2) Openstack-status shows the following:

                == Glance images ==
                Request returned failure status.
                Invalid OpenStack Identity credentials.
                ==Nova managed services ==
                ERROR: Unauthorized (HTTP 401)
                == Nova networks ==
                ERROR: Unauthorized (HTTP 401)
                == Nova instance flavors ==
                ERROR: Unauthorized (HTTP 401)
                == Nova instances ==
                ERROR: Unauthorized (HTTP 401)

Resolution:

Create New Services Project:

Create new "services" project/tenant via CLI (keystone tenant-create).
Obtain new "services" project/tenant ID via CLI (keystone tenant-list).

Determine NEW_SERVICES_ID:

Determine old project/tenant id of services project by looking at following users (nova,glance,neutron,heat,cinder) default_project_id in the user table of keystone database.   There default_project_id should all be the same and was the ID of the previous services project that was removed.


Edit MySQL Database:

use keystone;
update user set default_project_id="NEW_SERVICES_ID" where default_project_id="OLD_SERVICES_ID";
use ovs_neutron;
update networks set tenant_id="NEW_SERVICES_ID" where tenant_id="OLD_SERVICES_ID";
update subnets set tenant_id="NEW_SERVICES_ID" where tenant_id="OLD_SERVICES_ID";
update securitygroups set tenant_id="NEW_SERVICES_ID" where tenant_id="OLD_SERVICES_ID";