Complete Oracle VM Upgrade from 3.3.3 to 3.4.2

Oracle VM Upgrade

This document covers the steps to upgrade OVM environment from 3.3.3 to 3.4.2 (upgrade in one step without any incremental upgrades). This includes OVM manager server OS upgrade as well. Before proceeding to follow this document, check the compatibility of upgrade from current version to latest in Oracle documentation since it keeps on changing for each new version.

Upgrade Overview:
From a high level, the steps to upgrade Oracle VM are as follows:
1. Download required ISO from Oracle Edelivery
2. Backup OVM manager (OVMM)  and Oracle VM servers (OVS)
3. Upgrade OVMM to latest with runinstaller script
4. Reinstall OS on OVMM to latest version from ISO
5. Reinstall OVM manager with UUID and runinstaller script
6. Upgrade OVS via OVMM GUI one by one.


1. Download all necessary ISO from Edelivery
Download following ISO from Edelivery at https://edelivery.oracle.com/
a) Oracle VM Server 3.4.2 for x86 ISO
b) Oracle VM Server Source 3.4.2 ISO
c) Oracle VM Manager 3.4.2 ISO
d) Oracle Linux 7.3 ISO

2. Backup OVM manager and Oracle VM servers
All the backups generated should be saved in a safe location in your local computer and not on the OVMM or OVS servers.
a) To back up OVMM, first you should make a copy of the OVMM configuration file located at:  /u01/app/oracle/ovm-manager-3/.config
b) Next is manual backup of OVMM mysql database - This can be done by executing backup script below:
# /u01/app/oracle/ovm-manager-3/ovm_tools/bin/BackupDatabase -w
The backup generated will be stored at /u01/app/oracle/mysql/dbbackup/ManualBackup-xxxx
Save a copy of this folder and “OVMModelExport-xxx.xml” file available in the folder /u01/app/oracle/mysql/dbbackup/.
c) OS Configuration backup of OVMM and OVS server can be generated via “sosreport”. Execute the command “sosreport” in the server and save the sosreport file generated at the end.
d) If possible, take screenshots of each section and tabs in OVM manager GUI and save it.


3. Upgrade Oracle VM manager (OVMM) to latest with runinstaller script
a) Upload the OVMM 3.4.2 ISO to the OVMM server
b) Mount the ISO on /mnt
# mount –o loop /tmp/OVMM3.4.2-installer.iso /mnt
c) Access the “/mnt” folder and run the runinstaller script
    # ./runInstaller.sh
Choose option to “2” to “Upgrade”.

Enter required passwords in the next steps. Enter “1” to continue install.

Once install is complete, similar to below messages will appear.

Remove the temporary configuration file as mentioned above.
d) Try logging in to the manager GUI and verify the new version 3.4.2 from “Help  >> “About”. The new version should show and this completes OVMM upgrade.


4. Reinstall OS on OVMM to latest version from ISO
This document covers the upgrade of OL6.6 on OVMM server to OL 7.3 which is latest available at the time of upgrade. A direct upgrade from OL 6 to OL7 is not recommended even if an “In-place” upgrade is available. This is because the OVMM software is delicate and an in place upgrade on testing made it clear that OVMM software will be broken. Hence it is recommended to go with reinstall.

Note: A prerequisite for this re-install is ILOM access to OVMM server and java enabled browser.

a) Follow same steps as in point number “2. Backup OVM manager and Oracle VM servers” (2.a and 2.b) to backup OVMM before reinstalling the OS.
b) Access the ILOM of OVMM server
c) Launch the remote console. Launching remote console is only possible with Java 6 enabled browser.
d) Once console is launched, attach the OL 7.3 ISO to the servers from “Devices” >> CD-ROM Image
Select the file from local computer

e) Next it is time to reinstall server and hence we need to restart server from ILOM console >> Keyboard >> Control Alt Delete. This will send reboot signal to server
f) While server is starting to boot, keep on pressing F8 key to bring up the Boot Device selection screen.
g) Once a screen similar to below is shown, select CDROM option and it will boot from CD attached

h) Now OS installation steps will show. Enter server IP and details as old server and continue installation with all disks erased and space reclaimed.
i) Once install is completed, before rebooting server, remove CDROM attached from “Devices” >> CD-ROM Image. Uncheck the option. Then reboot.
j) The server should come up with new OL 7.3 OS. Login to server and verify IP and network configuration.


5. Reinstall OVM manager with UUID and runinstaller script
OVM manager can be restored at any point with UUID and runinstaller script.

a) Follow same steps as in point number “3. Upgrade Oracle VM manager (OVMM) to latest with runinstaller script”, but there is one difference in running runinstaller script as below
# ./runInstaller.sh --uuid  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
The UUID is obtained from backup we have already made for the file /u01/app/oracle/ovm-manager-3/.config.
Try to reuse old passwords and configuration and complete the reinstall
b) After re-installation stop the OVMM Command Line Interface, OVMM and the database before you restore the backup. On OVMM:
# /sbin/service ovmcli stop
# /sbin/service ovmm stop
# /sbin/service ovmm_mysql stop
c) Upload the “ManualBackup-xxxx “ folder created as per step “4. Reinstall OS on OVMM to latest version from ISO” (4.a) to the OVMM server under folder /u01/app/oracle/mysql/dbbackup
d) To initiate the database restore, as the oracle user, use the RestoreDatabase command located in /u01/app/oracle/ovm-manager-3/ovm_tools/bin, for example:
# chown –R oracle:dba ManualBackup-xxxx
# su – oracle
# cd /u01/app/oracle/mysql/dbbackup
# bash /u01/app/oracle/ovm-manager-3/ovm_tools/bin/RestoreDatabase.sh ManualBackup-xxxxx
e) Switch back to root user and run below:
# service ovmm_mysql start; service ovmm start; service ovmcli start
f) Run the following script to reconfigure the Oracle WebLogic Server and generate SSL key:

# export MW_HOME=/u01/app/oracle/Middleware

# /u01/app/oracle/ovm-manager-3/ovm_upgrade/bin/ovmkeytool.sh setupWebLogic

# /u01/app/oracle/ovm-manager-3/ovm_upgrade/bin/ovmkeytool.sh gensslkey
g) Restart Oracle VM Manager and then run the client certificate configuration script.

# /sbin/service ovmm restart

# /u01/app/oracle/ovm-manager-3/bin/configure_client_cert_login.sh

# /sbin/service ovmm restart
h) Access OVM manager GUI and refresh all Servers and VMs tab and perform a Refresh All on your existing server pools. Verify the manager GUI components, functioning and version

6. Upgrade Oracle VM servers (OVS) with OVMM GUI one by one.
It is always recommended to keep OVS servers and OVMM on same version for smooth functioning of clusters. OVS servers can be reinstalled completely and upgraded or an upgrade can be performed with OVMM GUI. This document covers OVS upgrade via OVMM GUI.

a) Create yum repository of OVS 3.4.2 in OVMM server. Select the OVS install ISO instead of CD for the same. Upload it in OVMM server and mount it on /mnt
# mount –o loop /tmp/OVS-3.4.2.iso /mnt
Following list of files should be available in the /mnt folder once ISO is mounted:
# ls -l /mnt/
drwxr-xr-x. 3 root root  2048 Sep 14  2016 EFI
-rw-r--r--. 1 root root  8791 Sep 14  2016 EULA
-rw-r--r--. 1 root root  8791 Sep 14  2016 eula.en_US
-rw-r--r--. 1 root root  3334 Sep 14  2016 eula.py
-rw-r--r--. 1 root root 17982 Sep 14  2016 GPL
drwxr-xr-x. 6 root root  2048 Sep 14  2016 grub2
drwxr-xr-x. 4 root root  2048 Sep 14  2016 images
drwxr-xr-x. 2 root root  2048 Sep 14  2016 isolinux
-rw-r--r--. 1 root root   103 Sep 14  2016 media.repo
drwxr-xr-x. 2 root root 94208 Sep 14  2016 Packages
lrwxrwxrwx. 1 root root    15 Sep 14  2016 repodata -> Server/repodata
-rw-r--r--. 1 root root  1011 Sep 14  2016 RPM-GPG-KEY
-rw-r--r--. 1 root root  1393 Sep 14  2016 RPM-GPG-KEY-ol5
-rw-r--r--. 1 root root  1011 Sep 14  2016 RPM-GPG-KEY-oracle
-rw-r--r--. 1 root root  1393 Sep 14  2016 RPM-GPG-KEY-oracle-ol5
drwxr-xr-x. 3 root root  2048 Sep 14  2016 Server
-rw-r--r--. 1 root root   106 Sep 14  2016 supportinfo
drwxr-xr-x. 3 root root  4096 Sep 14  2016 Transition
-r--r--r--. 1 root root  4439 Sep 14  2016 TRANS.TBL


b) Create a directory for repository access on OVMM server
# mkdir -p /var/www/repos
c) Copy the mounted ISO folder to the directory
# cp -arp /mnt/*  /var/www/repos/
d) Temporarily serve your repository with the Python SimpleHTTPServer module, as in the following example:
# cd /var/www
# python -m SimpleHTTPServer 80
Serving HTTP on 0.0.0.0 port 80 ...
e) Check that the repository is accessible from OVS server.
# wget http://ovmmip/repos/Server/repodata/repomd.xml
--2017-05-09 09:20:45--  http://192.168.0.205/repos/Server/repodata/repomd.xml
Connecting to 192.168.0.205:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3733 (3.6K) [text/xml]
Saving to: “repomd.xml”
100%[==========================================================================================================================================================>] 3,733       --.-K/s   in 0s     
2017-05-09 09:20:45 (185 MB/s) - “repomd.xml” saved [3733/3733]


f) After you set up the Yum repository, you must add it in the OVMM Web Interface.  Log in to the OVMM Web Interface. Either create a global server update group or a server update group for a specific server pool, as follows:
*Creating a global server update group
- Select the Reports and Resources tab.
- Select the Server Update Groups subtab.
- Expand the Server Update Groups folder, select GlobalX86ServerUpdateConfiguration.
- Expand the Server Update Groups folder, select GlobalX86ServerUpdateConfiguration, and then select Create New Server Update Repository.
- Create the server update group as in below screenshots

*Creating a server update group for a specific server pool
- Select the Servers and VMs tab.
- Expand the Server Pools folder and then select an appropriate server pool.
- Edit the server pool and select the Override Global Server Update Group option and then
select OK.
- Select the Server Update Repositories perspective and then select Create New Server
Update Repository.
- Create the server update group as in below screenshots

g) After you set up the Yum repositories with the content to upgrade OVS to Release 3.4 and
create the server update repository, you can perform the upgrade from the OVMM Web
Interface, as follows:
- Log in to the OVMM Web Interface. Select the Servers and VMs tab.
-  Expand the Server Pools folder and then select an appropriate server pool. Select the Servers perspective. Make sure there are no running VMs in the OVS server
- The Update Required column in the Servers perspective indicates “yes” if an update is available for each server within the server pool.

- Select each instance of OVS that you want to update and then select Update Server.
- Click OK when prompted to confirm the server upgrade.

- Each OVS is placed into maintenance mode and then upgraded. Any virtual machines
running on an instance of OVS are automatically migrated to another OVS when it is put into maintenance mode.
- Monitor the Update from OVS Message Log and OVMM Adminserver.log
- When the update is complete the OVS is restarted and remains in maintenance mode.
- Edit each instance of OVS to take it out of maintenance mode when the upgrade is complete.
h) Start the guests on the upgraded OVS and proceed upgrade on other OVS servers in pool

i) Refresh the File systems and repos, rediscover the OVS server if the OVMM is not showing correct status of the OVS. If still issue persist, discover the OVS server as a new one in OVM manager.
j) The pool VIP will be missing in OVM GUI. From OVM 3.4.1 versions, pool VIP is deprecated concept and hence you can remove the IP from monitoring in Nagios.



References:

OVMM upgrade overview:
http://docs.oracle.com/cd/E64076_01/E64078/html/vmiug-overview-upgrading.html

Backup OVMM:
http://docs.oracle.com/cd/E64076_01/E64083/html/vmadm-backup-restore-ovmm.html

Upgrade OVMM:
http://docs.oracle.com/cd/E64076_01/E64078/html/vmiug-manager-upgrading.html

Reinstall OS via ILOM:
https://www.youtube.com/watch?v=eqr2QdeESzU

Reinstall OVMM with UUID:
http://docs.oracle.com/cd/E64076_01/E64083/html/vmadm-manager-restore.html

Upgrade OVS:
http://docs.oracle.com/cd/E64076_01/E64078/html/vmiug-server-yum-upgrading.html