OPEN_NO_MASTER_KEY of AUTOLOGIN
After cloning a PDB with TDE into a RAC database, we saw OPEN_NO_MASTER_KEY in the second instance of the new PDB.
In most cases, it's because there's no master key was found, you have to set the master key in the password-protected keystore. But for a running RAC database, it's another issue.
Let's see the running RAC database.
1. Before Cloning
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 ORCLPDB READ WRITE NO
SQL> select inst_id, con_id, wallet_type, status from gv$encryption_wallet order by 1,2;
INST_ID CON_ID WALLET_TYPE STATUS
---------- ---------- -------------------- ------------------------------
1 1 AUTOLOGIN OPEN
1 2 AUTOLOGIN OPEN
1 3 AUTOLOGIN OPEN
2 1 AUTOLOGIN OPEN
2 2 AUTOLOGIN OPEN
2 3 AUTOLOGIN OPEN
As we can see, there's no problem with AUTOLOGIN.
2. Cloning
SQL> create pluggable database ORCLPDB2 from ORCLPDB storage unlimited tempfile reuse file_name_convert=('ORCLPDB','ORCLPDB2') keystore identified by "welcome1" parallel 16;
Pluggable database created.
SQL> alter pluggable database ORCLPDB2 open instances=all;
Pluggable database altered.
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 ORCLPDB READ WRITE NO
6 ORCLPDB2 READ WRITE NO
We open up the cloned PDB. So far, it looks good.
3. After Cloning
Let's check wallet status after cloning.
SQL> select inst_id, con_id, wallet_type, status from gv$encryption_wallet order by 1,2;
INST_ID CON_ID WALLET_TYPE STATUS
---------- ---------- -------------------- ------------------------------
1 1 AUTOLOGIN OPEN
1 2 AUTOLOGIN OPEN
1 3 AUTOLOGIN OPEN
1 6 AUTOLOGIN OPEN
2 1 AUTOLOGIN OPEN
2 2 AUTOLOGIN OPEN
2 3 AUTOLOGIN OPEN
2 6 AUTOLOGIN OPEN_NO_MASTER_KEY
8 rows selected.
OK, we do have AUTOLOGIN on the new PDB, but it seems that the wallet on node 2 is in a wrong status (OPEN_NO_MASTER_KEY). Is there any difference between the two nodes? Let's compare wallet in WALLET_ROOT at node 1 and 2.
Wallet at Node 1
[oracle@primary01 ~]$ ls -l $ORACLE_HOME/admin/ORCLCDB/wallet/tde
total 28
-rw------- 1 oracle asmadmin 5736 Jan 17 11:06 cwallet.sso
-rw------- 1 oracle asmadmin 2555 Aug 2 16:48 ewallet_2021080208481713.p12
-rw------- 1 oracle asmadmin 5467 Jan 17 11:06 ewallet_2022011703065723.p12
-rw------- 1 oracle asmadmin 5691 Jan 17 11:06 ewallet.p12
Wallet at Node 2
[oracle@primary01 ~]$ ssh primary02 ls -l $ORACLE_HOME/admin/ORCLCDB/wallet/tde
total 20
-rw------- 1 oracle oinstall 5512 Aug 2 16:48 cwallet.sso
-rw------- 1 oracle oinstall 2555 Aug 2 16:48 ewallet_2021080208481713.p12
-rw------- 1 oracle oinstall 5467 Aug 2 16:48 ewallet.p12
Oh, they're really different.
Solution
We should make the wallet identical among all nodes, so we copy wallet from node 1 to node 2.
[oracle@primary01 ~]$ scp -rp $ORACLE_HOME/admin/ORCLCDB/wallet/tde primary02:$ORACLE_HOME/admin/ORCLCDB/wallet/
ewallet.p12 100% 5691 425.6KB/s 00:00
cwallet.sso 100% 5736 10.0MB/s 00:00
ewallet_2021080208481713.p12 100% 2555 285.7KB/s 00:00
ewallet_2022011703065723.p12 100% 5467 5.4MB/s 00:00
[oracle@primary01 ~]$ ssh primary02 ls -l $ORACLE_HOME/admin/ORCLCDB/wallet/tde
total 28
-rw------- 1 oracle oinstall 5736 Jan 17 11:06 cwallet.sso
-rw------- 1 oracle oinstall 2555 Aug 2 16:48 ewallet_2021080208481713.p12
-rw------- 1 oracle oinstall 5467 Jan 17 11:06 ewallet_2022011703065723.p12
-rw------- 1 oracle oinstall 5691 Jan 17 11:06 ewallet.p12
Let's see the result.
SQL> select inst_id, con_id, wallet_type, status from gv$encryption_wallet order by 1,2;
INST_ID CON_ID WALLET_TYPE STATUS
---------- ---------- -------------------- ------------------------------
1 1 AUTOLOGIN OPEN
1 2 AUTOLOGIN OPEN
1 3 AUTOLOGIN OPEN
1 6 AUTOLOGIN OPEN
2 1 AUTOLOGIN OPEN
2 2 AUTOLOGIN OPEN
2 3 AUTOLOGIN OPEN
2 6 AUTOLOGIN OPEN
8 rows selected.
AUTOLOGIN is back to normal.
In this case, we perform a local cloning to demonstrate the phenomenon, actually, the solution is also workable to remote PDB cloning via a database link.
To overcome the issue thoroughly, you can set WALLET_ROOT in a shard space, e.g. moving wallet into ASM.