For some reason I could not delete a VM in my Openstack Cloud Platform. I tried to terminate the instance in the Horizon GUI but that did not work. I also tried to do it from the cli via:
# nova delete <uuid>
That did not work either. So I researched in on the web and noticed that the instances are kept in two places. The first place is in the nova directory:
/var/lib/nova/instances
[root@sdnnode2 instances]# ls
51426435-9f72-4077-91a4-6c89ea53894f 58fd794e-048a-4135-b064-17fb6bb3682b bbc44d49-62f7-4cdc-8e7f-564190c6f043 locks _base d0ce9950-752c-406f-91e0-3bd0ff0f488d snapshots
looking into those uuid directory names contained the following:
[root@sdnnode2 51426435-9f72-4077-91a4-6c89ea53894f]# ls
console.log disk libvirt.xml
However deleting these files still would not delete the VM from appearing in the Horizon GUI. These are the files that Openstack would use to spin up the VM.
The vm existed in MySQL.
I found a handy
sql_guide to brush me up on SQL commands. Then I had to access the SQL server with the correct credentials. If you don't remember it you can find it in the nova.conf file located in:
/etc/nova/nova.conf
[root@sdnnode2 nova]# cat nova.conf | grep sql
sql_connection = mysql://nova:
<mypassword>@10.161.38.4/nova
Then to access the SQL server:
[root@sdnnode2 nova]# sudo mysql -u nova -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 166
Server version: 5.1.73 Source distribution
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
show databases gives us.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| nova |
| test |
+--------------------+
mysql> use nova;
Database changed
mysql> show tables;
+--------------------------------------------+
| Tables_in_nova |
+--------------------------------------------+
| agent_builds |
| aggregate_hosts |
| aggregate_metadata |
| aggregates |
| block_device_mapping |
| bw_usage_cache |
<TRUNCATED>
| volume_usage_cache |
| volumes |
+--------------------------------------------+
115 rows in set (0.00 sec)
Here we can then find the instance uuid. First you could figure out the fields in a particular table:
mysql> describe key_pairs;
+-------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+----------------+
| created_at | datetime | YES | | NULL | |
| updated_at | datetime | YES | | NULL | |
| deleted_at | datetime | YES | | NULL | |
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(255) | YES | | NULL | |
| user_id | varchar(255) | YES | MUL | NULL | |
| fingerprint | varchar(255) | YES | | NULL | |
| public_key | mediumtext | YES | | NULL | |
| deleted | int(11) | YES | | NULL | |
+-------------+--------------+------+-----+---------+----------------+
9 rows in set (0.00 sec)
Then you can list the objects that are in the field you are searching.
mysql> select name from key_pairs;
+----------+
| name |
+----------+
| my-key |
| root-key |
+----------+
2 rows in set (0.00 sec)
I found out there are 5 tables you need to modify to get delete the VM and the should be done in this order:
security_group_instance_association
block_device_mapping
instance_info_caches
fixed_ips
instances
So I needed to find the uuid of the VM I was going to delete.
mysql> select uuid,hostname from instances where deleted=0;
+--------------------------------------+-----------+
| uuid | hostname |
+--------------------------------------+-----------+
| bbc44d49-62f7-4cdc-8e7f-564190c6f043 | right-vm1 |
| d0ce9950-752c-406f-91e0-3bd0ff0f488d | left-vm1 |
|
51426435-9f72-4077-91a4-6c89ea53894f |
test-vm |
| 58fd794e-048a-4135-b064-17fb6bb3682b | left-vm2 |
+--------------------------------------+-----------+
4 rows in set (0.00 sec)
I then started my process. I checked to see if the instance was not already marked deleted by checking for the condition of "deleted=0";
mysql> select instance_uuid from security_group_instance_association where deleted=0;
Empty set (0.00 sec)
I didn't have a security group association so I proceeded with the next step.
mysql> DELETE FROM block_device_mapping WHERE instance_uuid = "51426435-9f72-4077-91a4-6c89ea53894f";
Query OK, 1 row affected (0.00 sec)
I then followed the other steps
mysql> select instance_uuid from instance_info_caches where deleted=0;
DELETE FROM instance_info_caches WHERE instance_uuid = "51426435-9f72-4077-91a4-6c89ea53894f";
Query OK, 1 row affected (0.00 sec)
mysql>select instance_uuid from fixed_ips where deleted=0;
Empty set (0.00 sec)
I didn't have a fixed_ip so I went to the last step, which was to changed the "deleted" field.
You need to make sure the deleted number matches the id.
mysql> select id,deleted,uuid from instances;
+----+---------+--------------------------------------+
| id | deleted | uuid |
+----+---------+--------------------------------------+
| 31 | 31 | 028ff14b-55f2-4aa2-8fd9-37d70ff26a3b |
| 11 | 11 | 4c60dd0a-ae81-45e0-931b-c996e89b10d7 |
| 47 | 47 | 4ca2687f-35a4-4248-9777-24d9317e4c20 |
| 25 | 25 | 512747be-b608-423e-ac81-54f2ebcd7e58 |
| 35 | 0 | 51426435-9f72-4077-91a4-6c89ea53894f |
mysql> UPDATE instances SET deleted = 40 where uuid = "51426435-9f72-4077-91a4-6c89ea53894f";Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> select id,deleted,uuid from instances;
+----+---------+--------------------------------------+
| id | deleted | uuid |
+----+---------+--------------------------------------+
| 31 | 31 | 028ff14b-55f2-4aa2-8fd9-37d70ff26a3b |
| 11 | 11 | 4c60dd0a-ae81-45e0-931b-c996e89b10d7 |
| 47 | 47 | 4ca2687f-35a4-4248-9777-24d9317e4c20 |
| 25 | 25 | 512747be-b608-423e-ac81-54f2ebcd7e58 |
| 35 | 35 | 51426435-9f72-4077-91a4-6c89ea53894f |
Finally the instance was no longer in my Openstack server.
[root@sdnserver1 nova]# nova show 51426435-9f72-4077-91a4-6c89ea53894f
ERROR: No server with a name or ID of '51426435-9f72-4077-91a4-6c89ea53894f' exists.
