Oracle VM Manager 3.4.2: recuperación repositorio mysql corrupta
16/04/2018 -
En la versión 3.4.2 de Oracle Virtual Machine se ha detectado un bug de producto que consiste en el crecimiento desmesurado de la tabla en la BD mysql que contiene las estadísticas y, por extensión, a nivel del sistema operativo el fichero OVM_STATISTIC.ibd puede crecer hasta llenar el filesystem donde está instalado el producto. Este bug se descubrió al comprobar que no se podía acceder a la consola y que el filesystem /u01 estaba al 100%.
1
2
3
4
5
6
7
8
9
10
11
12
|
[root@ovmanager ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda3 11G 7.3G 1.4G 85% / devtmpfs 3.9G 0 3.9G 0% /dev tmpfs 3.7G 0 3.7G 0% /dev/shm tmpfs 3.7G 65M 3.6G 2% /run tmpfs 3.7G 0 3.7G 0% /sys/fs/cgroup /dev/xvda1 499M 105M 394M 22% /boot /dev/mapper/data-ovmm 25G 24G 0 100% /u01 tmpfs 594M 0 594M 0% /run/user/1000 tmpfs 594M 0 594M 0% /run/user/0 al buscar los ficheros de más de 1G en el filesystem |
1
2
3
4
|
[root@ovmanager u01]# find . -type f -size +1G ./app/oracle/mysql/data/ovs/OVM_STATISTIC.ibd [root@ovmanager u01]# ls -ltrh ./app/oracle/mysql/data/ovs/OVM_STATISTIC.ibd -rw-rw----. 1 oracle oinstall 13G Feb 15 01:56 ./app/oracle/mysql/data/ovs/OVM_STATISTIC.ibd |
Conectando a la BD, comprobamos el número de filas de la tabla:
1
2
3
4
5
6
7
8
9
|
mysql> select count(*) from OVM_STATISTIC; +----------+ | count(*) | +----------+ | 21528948 | +----------+ 1 row in set (40.76 sec) mysql> |
Para realizar la recuperación de la BD realizaremos los siguientes pasos:
- Parada de los servicios ovm
- Eliminación de la BD del OVM Manager
- Generación de nuevos certificados y arranque de los servicios
- Activación de los nuevos certificados
- Repopular la BD, redescubriendo los servidores
- Restaurar los “nombres simples” usando la siguiente nota de MOS: Restore OVM Manager “Simple Names” After a Rebuild/Reinstall (Doc ID 2129616.1)
Ejecución del procedimiento
1. Parada de los servicios OVM
1
2
3
|
[root@ovmanager ~]# service ovmm stop Stopping ovmm (via systemctl): [ OK ] [root@ovmanager ~]# |
2. Eliminación de la BD del OVM Manager
Consultamos el fichero /u01/app/oracle/ovm-manager-3/.config y obtenemos usuario y contraseña de conexión:
1
2
3
4
5
6
7
8
9
10
11
12
|
[root@ovmanager ~]# cat /u01/app/oracle/ovm-manager-3/.config DBTYPE=MySQL DBHOST=localhost SID=ovs <--- SID BBDD LSNR=49500 <--- puerto de la BD OVSSCHEMA=ovs <--- usuario de BD APEX=8080 WLSADMIN=weblogic OVSADMIN=admin COREPORT=54321 UUID=0004fb00000100006659158316ae6eae BUILDID=3.4.2.1384 |
Ejecutamos el siguiente comando basándonos en los datos previos:
1
|
sh /u01/app/oracle/ovm-manager-3/ovm_upgrade/bin/ovm_upgrade.sh --deletedb --dbuser=ovs --dbpass= --dbhost=localhost --dbport=49500 --dbsid=ovs |
La salida de la ejecución será parecido a lo siguiente:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
[root@ovmanager ~]# sh /u01/app/oracle/ovm-manager-3/ovm_upgrade/bin/ovm_upgrade.sh --deletedb --dbuser=ovs --dbpass=AromaCIS2017 --dbhost=localhost --dbport=49500 --dbsid=ovs COMMAND: /u01/app/oracle/ovm-manager-3/ovm_upgrade/bin/ovm_upgrade.sh --deletedb --dbuser=ovs --dbpass=AromaCIS2017 --dbhost=localhost --dbport=49500 --dbsid=ovs Copying deleted classes files to patch path location 2018-04-13 10:05:42,460 INFO Oracle OVM Manager Upgrade Processor 2018-04-13 10:05:42,460 INFO Apr 13, 2018 10:05:42 AM oracle.security.jps.JpsStartup start INFO: Jps initializing. Apr 13, 2018 10:05:43 AM oracle.security.jps.JpsStartup start INFO: Jps started. 2018-04-13 10:05:43,918 INFO Upgrade Initialization Starting 2018-04-13 10:05:43,979 INFO Oracle Distributed Object Fabric (ODOF): Copyright (C) 2007, 2016 Oracle. All rights reserved. 2018-04-13 10:05:43,979 INFO ODOF Version: 1.3.1.4 2018-04-13 10:05:43,979 INFO Initializing... 2018-04-13 10:05:44,264 INFO Initialization Complete 2018-04-13 10:05:44,265 INFO Upgrade Initialization Complete 2018-04-13 10:05:44,265 INFO Database Wipe Starting 2018-04-13 10:05:44,265 INFO Wiping Exchange 2018-04-13 10:05:44,276 INFO Initializing / Clearing Database Tables 2018-04-13 10:05:45,850 INFO Wiping Complete! 2018-04-13 10:05:45,850 INFO Database Wipe Complete |
Se completa el proceso de borrado. Comprobamos que se han liberado los 13G que ocupaba la tabla de estadísticas.
1
2
3
4
5
6
7
8
9
10
11
|
[root@ovmanager ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda3 11G 7.3G 1.4G 85% / devtmpfs 3.9G 0 3.9G 0% /dev tmpfs 3.7G 0 3.7G 0% /dev/shm tmpfs 3.7G 65M 3.6G 2% /run tmpfs 3.7G 0 3.7G 0% /sys/fs/cgroup /dev/xvda1 499M 105M 394M 22% /boot /dev/mapper/data-ovmm 25G 11G 13G 45% /u01 tmpfs 594M 0 594M 0% /run/user/1000 tmpfs 594M 0 594M 0% /run/user/0 |
3. Generación de nuevos certificados y arranque los servicios
service ovmm start
export MW_HOME=/u01/app/oracle/Middleware
/u01/app/oracle/ovm-manager-3/ovm_upgrade/bin/ovmkeytool.sh setupWebLogic
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
[root@ovmanager ~]# service ovmm start Starting ovmm (via systemctl): [ OK ] [root@ovmanager ~]# export MW_HOME=/u01/app/oracle/Middleware [root@ovmanager ~]# /u01/app/oracle/ovm-manager-3/ovm_upgrade/bin/ovmkeytool.sh setupWebLogic Apr 13, 2018 10:11:03 AM oracle.security.jps.JpsStartup start INFO: Jps initializing. Apr 13, 2018 10:11:04 AM oracle.security.jps.JpsStartup start INFO: Jps started. Updating keystore information in WebLogic Oracle MiddleWare Home (MW_HOME): [/u01/app/oracle/Middleware] WebLogic domain directory: [/u01/app/oracle/ovm-manager-3/domains/ovm_domain] WebLogic server name: [AdminServer] WebLogic username: [weblogic] WebLogic password: [********] WLST session logged at: /tmp/wlst-session5269642832261808000.log |
4. Activación de los nuevos certificados
/u01/app/oracle/ovm-manager-3/bin/configure_client_cert_login.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
|
2018-04-13 10:29:15,333 [main] INFO ovm.wlst.domainbuilder.Domain - Trying to connect to t3://localhost:7001... 2018-04-13 10:29:26,673 [main] INFO ovm.wlst.domainbuilder.Domain - Trying to connect to t3://localhost:7001... 2018-04-13 10:29:27,602 [main] INFO ovm.wlst.domainbuilder.Domain - Connected. 2018-04-13 10:29:27,627 [main] INFO ovm.wlst.domainbuilder.Domain - AdminServer state is RUNNING 2018-04-13 10:29:27,628 [main] INFO ovm.wlst.commands - Deploying ovm_core Deploying application from /u01/app/oracle/ovm-manager-3/ovm_wlst/deploy/ovm_core/app/ovm_core_3.4.2.1384.ear to targets AdminServer (upload=false) ... ...Completed the deployment of Application with status completed Current Status of your Deployment: Deployment command type: deploy Deployment State : completed Deployment Message : no message Already in Domain Config Tree Already in Domain Config Tree 2018-04-13 10:29:37,908 [main] INFO ovm.wlst.domainbuilder.Domain - Created a user named appframework SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/u01/app/oracle/ovm-manager-3/ovm_cli/lib/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/u01/app/oracle/Middleware/wlserver/modules/features/weblogic.server.merged.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. Apr 13, 2018 10:29:38 AM oracle.security.jps.JpsStartup start INFO: Jps initializing. Apr 13, 2018 10:29:40 AM oracle.security.jps.JpsStartup start INFO: Jps started. 2018-04-13 10:30:33,748 [main] INFO com.oracle.appfw.ovm.ws.client.KeytoolHelper - Writing cacert.pem 2018-04-13 10:30:33,750 [main] INFO com.oracle.appfw.ovm.ws.client.KeytoolHelper - Importing cacert.pem with alias ovmca 2018-04-13 10:30:34,061 [main] INFO com.oracle.appfw.ovm.ws.client.KeytoolHelper - Generating key pair for appframework in /u01/app/oracle/ovm-manager-3/domains/ovm_domain/security/ovmclient.jks 2018-04-13 10:30:34,924 [main] INFO com.oracle.appfw.ovm.ws.client.KeytoolHelper - Exporting clientcert.pem from /u01/app/oracle/ovm-manager-3/domains/ovm_domain/security/ovmclient.jks 2018-04-13 10:30:35,065 [main] INFO com.oracle.appfw.ovm.ws.client.KeytoolHelper - Reading clientcert.pem 2018-04-13 10:30:35,066 [main] INFO com.oracle.appfw.ovm.ws.client.SSLClientUtil - Signing certificate 2018-04-13 10:30:35,190 [main] INFO com.oracle.appfw.ovm.ws.client.KeytoolHelper - Writing clientcert.pem 2018-04-13 10:30:35,191 [main] INFO com.oracle.appfw.ovm.ws.client.KeytoolHelper - Importing clientcert.pem with alias appframework 2018-04-13 10:30:35,667 [main] INFO com.oracle.appfw.ovm.ws.client.SSLClientUtil - Deleting cacert.pem 2018-04-13 10:30:35,667 [main] INFO com.oracle.appfw.ovm.ws.client.SSLClientUtil - Deleting clientcert.pem 2018-04-13 10:30:35,696 [main] INFO ovm.wlst.commands - Deploying ovm_console Deploying application from /u01/app/oracle/ovm-manager-3/ovm_wlst/deploy/ovm_console/app/ovm_console_3.4.2.1384.ear to targets AdminServer (upload=false) ... ....Completed the deployment of Application with status completed Current Status of your Deployment: Deployment command type: deploy Deployment State : completed Deployment Message : no message Client certificate login configuration complete |
Reiniciamos el servicio para aplicar los certificados nuevos.
1
2
3
4
|
[root@ovmanager ~]# service ovmm stop Stopping ovmm (via systemctl): [ OK ] [root@ovmanager ~]# service ovmm start Starting ovmm (via systemctl): [ OK ] |
5. Repopulamos la BD redescubriendo los OV Servers
Descubrimos los OV Servers para repopular la BD. Para ello conectamos a la consola, ya accesible.
Tras redescubrirlos, vemos que los nombres son genéricos y ya no están los que había previamente. Se debe a que no se han cargado los “nombres sencillos” (o friendly) que se almacenan en un fichero xml del dominio.
6. Restaurar los “nombres simples”
En MOS, se indica que para recuperar los “nombres sencillos” de los componentes es necesario seguir los pasos de la nota Restore OVM Manager “Simple Names” After a Rebuild/Reinstall (Doc ID 2129616.1). Sin embargo, tras realizar un refresco de los repositorios después de descubrir los servidores hemos recuperado los “nombres sencillos” de los componentes.
Por lo que con esta última acción damos por finalizado el procedimiento de recuperación de la BD, ya que no se ve necesario ejecutar ninguna acción adicional.