This is a step by step demonstration on how to migrate any ASM disk groups from a cluster to another. May be use, with or without virtualization and may be used with storage layer snapshot for fast environment provisioning.

Step 01 – Shutdown source database(s) on VMWARE servers

Shutdown all databases hosted in the targeted Disk groups for which you want consistency. Then unmount the disk groups.

$ORACLE_HOME/bin/srvctl stop database -db cdb001

$ORACLE_HOME/bin/asmcmd umount FRA

$ORACLE_HOME/bin/asmcmd umount DATA

Step 02 – Re route LUNs from the storage array to newf servers

Create a snapshot and make the snapshot LUNs visible for Oracle Virtual Server (OVS) according the third-party storage technology.

Step 03 – Add LUNs to DomUs (VMs)

Then, we refresh the storage layer from OVM Manager to present LUNs in each OVS

OVM> refresh storagearray name=STORAGE_ARRAY_01

Step 04 – Then, tell OVM Manager to add LUNs to the VMs in which we want our databases to be migrated

create VmDiskMapping slot=20 physicalDisk=sa01_clus01_asm_data01 name=sa01_clus01_asm_data01 on Vm name=rac001
create VmDiskMapping slot=21 physicalDisk=sa01_clus01_asm_data02 name=sa01_clus01_asm_data02 on Vm name=rac001
create VmDiskMapping slot=22 physicalDisk=sa01_clus01_asm_data03 name=sa01_clus01_asm_data03 on Vm name=rac001
create VmDiskMapping slot=23 physicalDisk=sa01_clus01_asm_data04 name=sa01_clus01_asm_data04 on Vm name=rac001
create VmDiskMapping slot=24 physicalDisk=sa01_clus01_asm_data05 name=sa01_clus01_asm_data05 on Vm name=rac001
create VmDiskMapping slot=25 physicalDisk=sa01_clus01_asm_data06 name=sa01_clus01_asm_data06 on Vm name=rac001
create VmDiskMapping slot=26 physicalDisk=sa01_clus01_asm_reco01 name=sa01_clus01_asm_reco01 on Vm name=rac001
create VmDiskMapping slot=27 physicalDisk=sa01_clus01_asm_reco02 name=sa01_clus01_asm_reco02 on Vm name=rac001

create VmDiskMapping slot=20 physicalDisk=sa01_clus01_asm_data01 name=sa01_clus01_asm_data01 on Vm name=rac002
create VmDiskMapping slot=21 physicalDisk=sa01_clus01_asm_data02 name=sa01_clus01_asm_data02 on Vm name=rac002
create VmDiskMapping slot=22 physicalDisk=sa01_clus01_asm_data03 name=sa01_clus01_asm_data03 on Vm name=rac002
create VmDiskMapping slot=23 physicalDisk=sa01_clus01_asm_data04 name=sa01_clus01_asm_data04 on Vm name=rac002
create VmDiskMapping slot=24 physicalDisk=sa01_clus01_asm_data05 name=sa01_clus01_asm_data05 on Vm name=rac002
create VmDiskMapping slot=25 physicalDisk=sa01_clus01_asm_data06 name=sa01_clus01_asm_data06 on Vm name=rac002
create VmDiskMapping slot=26 physicalDisk=sa01_clus01_asm_reco01 name=sa01_clus01_asm_reco01 on Vm name=rac002
create VmDiskMapping slot=27 physicalDisk=sa01_clus01_asm_reco02 name=sa01_clus01_asm_reco02 on Vm name=rac002

At this stage we have all LUNs of our both disk groups for DATA and FRA available on both nodes of the cluster.

Step 05 – Migrate disk in AFD

We can rename disk groups if required or if a disk group with the same name already exists

renamedg phase=both dgname=DATA newdgname=DATAMIG verbose=true asm_diskstring='/dev/xvdr1','/dev/xvds1','/dev/xvdt1','/dev/xvdu1','/dev/xvdv1','/dev/xvdw1'
renamedg phase=both dgname=FRA  newdgname=FRAMIG  verbose=true asm_diskstring='/dev/xvdx1','/dev/xvdy1'

Then we migrate disks into AFD configuration

$ORACLE_HOME/bin/asmcmd afd_label DATAMIG /dev/xvdr1 --migrate
$ORACLE_HOME/bin/asmcmd afd_label DATAMIG /dev/xvds1 --migrate
$ORACLE_HOME/bin/asmcmd afd_label DATAMIG /dev/xvdt1 --migrate
$ORACLE_HOME/bin/asmcmd afd_label DATAMIG /dev/xvdu1 --migrate
$ORACLE_HOME/bin/asmcmd afd_label DATAMIG /dev/xvdv1 --migrate
$ORACLE_HOME/bin/asmcmd afd_label DATAMIG /dev/xvdw1 --migrate
$ORACLE_HOME/bin/asmcmd afd_label FRAMIG  /dev/xvdx1 --migrate
$ORACLE_HOME/bin/asmcmd afd_label FRAMIG  /dev/xvdy1 --migrate

Step 06 – Mount disk groups on the new cluster and add database(s) in the cluster

$ORACLE_HOME/bin/asmcmd mount DATAMIG
$ORACLE_HOME/bin/asmcmd mount FRAMIG

Then add database(s) to cluster (repeat for each database)

$ORACLE_HOME/bin/srvctl add database -db cdb001 \
-oraclehome /u01/app/oracle/product/12.2.0/dbhome_1 \
-dbtype RAC \
-spfile +DATAMIG/CDB001/spfileCDB001.ora \
-diskgroup DATAMIG,FRAMIG

Step 06 – Startup database

In that case, we renamed the disk groups so we need to modify file locations and some parameter values

create pfile='/tmp/initcdb001.ora' from spfile='+DATAMIG/<spfile_path>' ;
-- modify controlfiles, recovery area and any other relevant paramters
create spfile='+DATAMIG/CDB001/spfileCDB001.ora' from pfile='/tmp/initcdb001.ora' ;

ALTER DATABASE RENAME FILE '+DATA/<datafile_paths>','+DATAMIG/<datafile_paths>'
ALTER DATABASE RENAME FILE '+DATA/<tempfile_paths>','+DATAMIG/<tempfile_paths>'
ALTER DATABASE RENAME FILE '+DATA/<onlinelog_paths>','+DATAMIG/<onlinelog_paths>'
ALTER DATABASE RENAME FILE '+FRA/<onlinelog_paths>', '+FRAMIG/<onlinelog_paths>'

Then start the database

$ORACLE_HOME/bin/srvctl start database -db cdb001

This method can be used to easily migrated TB of data with almost no pain, reducing at most as possible the downtime period. For near Zero downtime migration, just add a GoldenGate replication on top of that.

The method describes here is also perfectly applicable for ASM snapshot in order to duplicate huge volume from one environment to another. This permits fast environment provisioning without the need to duplicate data over the network nor impact storage layer with intensive I/Os.

I hope it may help and please do not hesitate to contact us if you have any questions or require further information.

Cet article Migrate Oracle Database(s) and ASM diskgroups from VMWARE to Oracle VM est apparu en premier sur Blog dbi services.

When ODA Lite was introduced, with ODA X6-2 S/M/L, and now with ODA x7-2 S/M, a new ‘odacli’ was there to manage it. It will probably replace the oakcli for ODA HA as well in the future. One big difference: it uses a repository to record the configuration and the operations. I don’t really like it because when something fails you are blocked. Oracle Support can modify the directory, but they ask for an access to the machine for that and this is not easy in secured environments. Anyway, I really don’t understand why another repository has been introduced. We already have the Oracle Inventory, the Grid Infrastructure resources, the Linux /etc files,… And now we have a closed repository which controls everything, accessible only with the very limited odacli commands which are not the best example of automation code and error handling.

This post is about viewing what is inside. You may also want to update it in case you have a problem. I can’t tell you not to do it: this blog has readers who fixed critical issues by editing the binary data in system files, so changing some metadata in an embedded SQL database is not so dangerous. On the other hand, you take the risk to mess up everything. So better contact Oracle Support of you are not 142% sure about what you do. But when the support is long to answer, asks a remote access, or has no other solution than re-image the ODA, you may have to find other alternatives. Just limit yourseld to what you know you can do without risk.

So, this repository is stored in an embedded JavaDB which is, as far as I understand it, An Apache Derby database recolored in red by Oracle. There’s a jdbc driver to access it.

You find the repository on your ODA in the following directory:
/opt/oracle/dcs/repo

You will probably copy the directory elsewhere to look at it. And you may do that with the DCS agent stopped.

SQuirreL SQL Client

I used SQuirreL SQL Client to read this database:

Download SQuirreL SQL Client: squirrel-sql-snapshot-20180206_2232-standard.jar from https://sourceforge.net/projects/squirrel-sql/files/latest/download and install it.

Download derby.jar from https://db.apache.org/derby/derby_downloads.html

Run SQuirreL SQL Client, and add the derby.jar:

Connect to it. If you still have the repo at the original place, the URL is jdbc:derby:/opt/oracle/dcs/repo/node_0. There is no user and no password.

Then, in the ‘APP’ catalog, you can browse the tables:

SchemaSpy

You probably want to see the data model for those few tables. I did it on a 12.1.2.11.0 repository. I used SchemaSpy (http://schemaspy.org/) which is awesome because it uses the awesome Graphviz (https://www.graphviz.org/) for the visual representation. If you want to do the same, here is how I did it:


export PATH=$PATH:"/cygdrive/c/Program Files (x86)/Graphviz2.38/bin"
java -jar schemaspy*.jar -t derby -db ./repo/node_0 -dp ./derby.jar -o ODAviz -u "" -cat "APP"


Here are some of the schemas generated if you want to have a look at what is stored in the ODA repository: ODAviz.pub.zip

The schema is very simple. Only a few referential integrity constraints and very simple information.

One additional warning: modifications here are not supported by Oracle, and that may even be forbidden as the Capacity On Demand core count is also stored there.

Cet article ODA Lite: What is this ‘odacli’ repository? est apparu en premier sur Blog dbi services.

Oracle 18c is out, in the Oracle Cloud, and the first thing I do with a new version is testing how long it takes to upgrade a previous version PDB by unplug/plug. Faster upgrade should be the benefit of having a slim dictionary where the system objects are reduced to metadata links and data links. However, it looks like upgrading the PDB dictionary still takes the same time as upgrading the CDB$ROOT.

The idea is to create a DBaaS service with a new CDB in 18.1 and plug a PDB coming from 12.2.0.1. Actually, I’m saying 18.1 but that may be 18.0 as I’m now lost in those version numbers. The cloud service was created with version: “18.0.0.0”, V$VERSION displays 18.1.0.0 for the release and 18.0.0.0 for the version:
Connected to:
Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.1.0.0.0
My understanding is that the 18.0.0.0 is the version of the 18c dictionary, which will need a full upgrade only for 19c (19.0.0.0). And 18.1.0.0 is about the version, which will be incremented by Release Updates later.

I have an unplugged PDB that I plug into the new CDB:
SQL> create pluggable database PDB0 using '/u01/app/temp/PDB0.pdb';
Pluggable database PDB0 created.

When I open it, I get a warning:
SQL> alter pluggable database pdb0 open;
 
ORA-24344: success with compilation error
24344. 00000 - "success with compilation error"
*Cause: A sql/plsql compilation error occurred.
*Action: Return OCI_SUCCESS_WITH_INFO along with the error code
 
Pluggable database PDB0 altered.


Then I check the PDB PLUG IN VIOLATIONS:
 
SQL> select * from pdb_plug_in_violations;
TIME NAME CAUSE TYPE ERROR_NUMBER LINE MESSAGE STATUS ACTION CON_ID
---- ---- ----- ---- ------------ ---- ------- ------ ------ ------
24-FEB-18 08.35.16.965295000 PM PDB0 OPTION ERROR 0 1 Database option APS mismatch: PDB installed version 12.2.0.1.0. CDB installed version 18.0.0.0.0. PENDING Fix the database option in the PDB or the CDB 1
24-FEB-18 08.35.16.966343000 PM PDB0 OPTION ERROR 0 2 Database option CATALOG mismatch: PDB installed version 12.2.0.1.0. CDB installed version 18.0.0.0.0. PENDING Fix the database option in the PDB or the CDB 1
24-FEB-18 08.35.16.966556000 PM PDB0 OPTION ERROR 0 3 Database option CATJAVA mismatch: PDB installed version 12.2.0.1.0. CDB installed version 18.0.0.0.0. PENDING Fix the database option in the PDB or the CDB 1
24-FEB-18 08.35.16.966780000 PM PDB0 OPTION ERROR 0 4 Database option CATPROC mismatch: PDB installed version 12.2.0.1.0. CDB installed version 18.0.0.0.0. PENDING Fix the database option in the PDB or the CDB 1
24-FEB-18 08.35.16.966940000 PM PDB0 OPTION ERROR 0 5 Database option CONTEXT mismatch: PDB installed version 12.2.0.1.0. CDB installed version 18.0.0.0.0. PENDING Fix the database option in the PDB or the CDB 1
24-FEB-18 08.35.16.967096000 PM PDB0 OPTION ERROR 0 6 Database option DV mismatch: PDB installed version 12.2.0.1.0. CDB installed version 18.0.0.0.0. PENDING Fix the database option in the PDB or the CDB 1
24-FEB-18 08.35.16.967250000 PM PDB0 OPTION ERROR 0 7 Database option JAVAVM mismatch: PDB installed version 12.2.0.1.0. CDB installed version 18.0.0.0.0. PENDING Fix the database option in the PDB or the CDB 1
24-FEB-18 08.35.16.967403000 PM PDB0 OPTION ERROR 0 8 Database option OLS mismatch: PDB installed version 12.2.0.1.0. CDB installed version 18.0.0.0.0. PENDING Fix the database option in the PDB or the CDB 1
24-FEB-18 08.35.16.967602000 PM PDB0 OPTION ERROR 0 9 Database option ORDIM mismatch: PDB installed version 12.2.0.1.0. CDB installed version 18.0.0.0.0. PENDING Fix the database option in the PDB or the CDB 1
24-FEB-18 08.35.16.967785000 PM PDB0 OPTION ERROR 0 10 Database option OWM mismatch: PDB installed version 12.2.0.1.0. CDB installed version 18.0.0.0.0. PENDING Fix the database option in the PDB or the CDB 1
24-FEB-18 08.35.16.967939000 PM PDB0 OPTION ERROR 0 11 Database option SDO mismatch: PDB installed version 12.2.0.1.0. CDB installed version 18.0.0.0.0. PENDING Fix the database option in the PDB or the CDB 1
24-FEB-18 08.35.16.968091000 PM PDB0 OPTION ERROR 0 12 Database option XDB mismatch: PDB installed version 12.2.0.1.0. CDB installed version 18.0.0.0.0. PENDING Fix the database option in the PDB or the CDB 1
24-FEB-18 08.35.16.968246000 PM PDB0 OPTION ERROR 0 13 Database option XML mismatch: PDB installed version 12.2.0.1.0. CDB installed version 18.0.0.0.0. PENDING Fix the database option in the PDB or the CDB 1
24-FEB-18 08.35.16.968398000 PM PDB0 OPTION ERROR 0 14 Database option XOQ mismatch: PDB installed version 12.2.0.1.0. CDB installed version 18.0.0.0.0. PENDING Fix the database option in the PDB or the CDB 1
24-FEB-18 08.35.16.971138000 PM PDB0 Parameter WARNING 0 1 CDB parameter compatible mismatch: Previous '12.2.0' Current '18.0.0' PENDING Please check the parameter in the current CDB 1
24-FEB-18 08.35.17.115346000 PM PDB0 VSN not match ERROR 0 1 PDB's version does not match CDB's version: PDB's version 12.2.0.1.0. CDB's version 18.0.0.0.0. PENDING Either upgrade the PDB or reload the components in the PDB. 4


The messages are clear: all components have a 12.2.0.1 dictionary and must be upgraded to a 18.0.0.0.0 one

The PDB is opened in MIGRATE mode with only RESTRICTED sessions enabled:
SQL> show pdbs
SP2-0382: The SHOW PDBS command is not available.
SQL> pdbs
 
CON_ID CON_NAME OPEN_MODE RESTRICTED STATUS FOREIGN_CDB_DBID FOREIGN_PDB_ID CREATION_TIME REFRESH_MODE REFRESH_INTERVAL LAST_REFRESH_SCN CDB_DBID CURRENT_SCN
------ -------- --------- ---------- ------ ---------------- -------------- ------------- ------------ ---------------- ---------------- -------- -----------
2 PDB$SEED READ ONLY NO NORMAL 1201448 2 11:42:25 NONE 942476327 3958500
3 CDB1PDB MOUNTED NORMAL 942476327 2 19:58:55 NONE 942476327 3958500
4 PDB0 MIGRATE YES NEW 941386968 3 20:34:50 NONE 942476327 3958500


Then, here is the upgrade for this newly plugged PDB0:

[oracle@DBaaS18c 18c]$ dbupgrade -c PDB0
 
Argument list for [/u01/app/oracle/product/18.0.0/dbhome_1/rdbms/admin/catctl.pl] Run in c = PDB0
Do not run in C = 0
Input Directory d = 0
Echo OFF e = 1
Simulate E = 0
Forced cleanup F = 0
Log Id i = 0
Child Process I = 0
Log Dir l = 0
Priority List Name L = 0
Upgrade Mode active M = 0
SQL Process Count n = 0
SQL PDB Process Count N = 0
Open Mode Normal o = 0
Start Phase p = 0
End Phase P = 0
Reverse Order r = 0
AutoUpgrade Resume R = 0
Script s = 0
Serial Run S = 0
RO User Tablespaces T = 0
Display Phases y = 0
Debug catcon.pm z = 0
Debug catctl.pl Z = 0
 
catctl.pl VERSION: [18.0.0.0.0] STATUS: [Production] BUILD: [RDBMS_18.1.0.0.0_LINUX.X64_180103.1] ...
The Build number mentions 18.1 built on 03-JAN-2018

Look at the summary to see the time it takes:
Oracle Database Release 18 Post-Upgrade Status Tool 02-24-2018 21:36:5
[PDB0]  
Component Current Full Elapsed Time
Name Status Version HH:MM:SS
 
Oracle Server UPGRADED 18.1.0.0.0 00:13:37
JServer JAVA Virtual Machine UPGRADED 18.1.0.0.0 00:00:51
Oracle XDK UPGRADED 18.1.0.0.0 00:00:21
Oracle Database Java Packages UPGRADED 18.1.0.0.0 00:00:05
OLAP Analytic Workspace UPGRADED 18.1.0.0.0 00:00:11
Oracle Label Security UPGRADED 18.1.0.0.0 00:00:03
Oracle Database Vault UPGRADED 18.1.0.0.0 00:00:34
Oracle Text UPGRADED 18.1.0.0.0 00:00:11
Oracle Workspace Manager UPGRADED 18.1.0.0.0 00:00:18
Oracle Real Application Clusters UPGRADED 18.1.0.0.0 00:00:00
Oracle XML Database UPGRADED 18.1.0.0.0 00:00:49
Oracle Multimedia UPGRADED 18.1.0.0.0 00:01:03
Spatial UPGRADED 18.1.0.0.0 00:02:06
Oracle OLAP API UPGRADED 18.1.0.0.0 00:00:08
Upgrade Datapatch 00:00:05
Final Actions 00:00:09
Post Upgrade 00:00:02
Post Upgrade Datapatch 00:00:04
 
Total Upgrade Time: 00:20:47 [PDB0]  
Database time zone version is 26. It is older than current release time
zone version 31. Time zone upgrade is needed using the DBMS_DST package.
 
Grand Total Upgrade Time: [0d:0h:21m:10s]

Here we see 18.1 but the important number is the time: 21 minutes… Once again, I see no improvement in the time to upgrade the PDB dictionary. This was on a service with 2 OCPU and I’ve run a whole CDB upgrade with a similar shape and the time to upgrade the CDB$ROOT is exaclty the same – see the screenshot on the right.

Finally I open the PDB:

SQL> alter pluggable database pdb0 open;
Pluggable database PDB0 altered.


And check that the violations are resolved:

SQL> select * from pdb_plug_in_violations where status'RESOLVED';
 
TIME NAME CAUSE TYPE ERROR_NUMBER LINE MESSAGE STATUS ACTION CON_ID
---- ---- ----- ---- ------------ ---- ------- ------ ------ ------
24-FEB-18 09.46.25.302228000 PM PDB0 OPTION WARNING 0 15 Database option RAC mismatch: PDB installed version 18.0.0.0.0. CDB installed version NULL. PENDING Fix the database option in the PDB or the CDB 4

Ok, I suppose I can ignore that as this is not RAC.

I’ve not seen a lot of differences in the dbupgrade output. There’s a new summary of versions before and after upgrade, which was not there in 12c:

DOC>#####################################################
DOC>#####################################################
DOC>
DOC> DIAG OS Version: linux x86_64-linux-thread-multi 2.6.39-400.211.1.el6uek.x86_64
DOC> DIAG Database Instance Name: CDB1
DOC> DIAG Database Time Zone Version: 31
DOC> DIAG Database Version Before Upgrade: 12.2.0.1.0
DOC> DIAG Database Version After Upgrade: 18.1.0.0.0
DOC>#####################################################
DOC>#####################################################


However, be careful with this information. The OS Version is not correct:

[opc@DB ~]$ uname -a
Linux DB 4.1.12-112.14.10.el6uek.x86_64 #2 SMP Mon Jan 8 18:24:01 PST 2018 x86_64 x86_64 x86_64 GNU/Linux

It seems that this info comes from Config.pm which is the OS version where the perl binaries were built…

In summary, nothing changes here about the time it takes to upgrade a PDB when plugged into a new CDB.
However, in 18c (and maybe only with next Release Updates) we should have a way to get this improved by recording the upgrade of CDB$ROOT and re-playing a trimmed version on the PDB dictionaries, in the same way as in Application Containers for application upgrades. We already see some signs of it with ‘_enable_cdb_upgrade_capture’ undocumented parameter and PDB_UPGRADE_SYNC database property. This may even become automatic when PDB is opened with the PDB_AUTO_UPGRADE property. But that’s for the future, and not yet documented.

For the moment, you still need to run a full catupgrd on each container, through catctl.pl called by the ‘dbupgrade’ script. Here on a 2 OCPU service, it takes 20 minutes.

Cet article PDB upgrade from 12c to 18c est apparu en premier sur Blog dbi services.

In a previous post I’ve listed several free online services which run an Oracle XE so that you can test your SQL easily. You may want use Oracle XE further, with full access to the database and its host, and still from a web browser. You probably have a Google account. Then you also have a Virtual Machine on the Google Cloud (0.5 vCPU / 1.70 GB RAM boostable to 1 vCPU / 3.75 GB) and 5 GB of persistent storage (as long as you used it in the 120 previous days). Just try this Google Cloud Shell: https://console.cloud.google.com/cloudshell.
In this post, I explain how to install Oracle XE there.

First, you need to download Oracle XE. You do that on your laptop to upload it to the Google Cloud Shell. For legal reason, there is no automated way to download it with wget because you have to manually accept the OTN License Term: http://www.oracle.com/technetwork/database/database-technologies/express-edition/downloads/index.html and choose ‘Oracle Database Express Edition 11g Release 2 for Linux x64′

You can try to upload it to the Cloud Shell directly (menu on top right – upload file) but I had problems with the size of the file, so I’ve split it into two files:

split -b 150M oracle-xe-11.2.0-1.0.x86_64.rpm.zip

You should have ‘split’ even on Windows (Ubuntu Bash Shell) but you can also use any tool. 7-zip can do that.

I uploaded the two files:

Now on the Google Cloud shell, concatenate the files back to the .zip:

franck_pachot@cloudshell:~$ cat xa* > oracle-xe-11.2.0-1.0.x86_64.rpm.zip


Unzip it:

franck_pachot@cloudshell:~$ unzip oracle-xe-11.2.0-1.0.x86_64.rpm.zip
Archive: ora.zip
creating: Disk1/
creating: Disk1/upgrade/
inflating: Disk1/upgrade/gen_inst.sql
creating: Disk1/response/
inflating: Disk1/response/xe.rsp
inflating: Disk1/oracle-xe-11.2.0-1.0.x86_64.rpm


This .zip contains a .rpm but we are on Debian in the Google Cloud Shell. In addition to that, I’ll not follow the standard installation of Oracle XE because only my $HOME filesystem is persistent, so I want everything there. I need rpm2cpio to extract from the .rpm, and I’ll need libaio1 to install Oracle:

franck_pachot@cloudshell:~$ sudo apt-get -y install rpm2cpio libaio1


Here is the extraction:

franck_pachot@cloudshell:~$ rpm2cpio Disk1/oracle-xe-11.2.0-1.0.x86_64.rpm | cpio -idmv


This extracted to u01, etc and usr in my $HOME directory and I’ll leave the Oracle Home there.
I can remove the intermediate files:

franck_pachot@cloudshell:~$ rm -f xa? oracle-xe-11.2.0-1.0.x86_64.rpm.zip Disk1


The Oracle XE deployment contains a ‘createdb.sh’ which will create the XE database. You don’t have dbca here, you don’t have templates. Oracle XE is build to be small.
Just set ORACLE_HOME, PATH, LD_LIBRARY_PATH and run createdb.sh

export ORACLE_HOME=$HOME/u01/app/oracle/product/11.2.0/xe
echo "$PATH" | grep "$ORACLE_HOME" || export PATH=$PATH:$ORACLE_HOME/bin
echo "$LD_LIBRARY_PATH" | grep "$ORACLE_HOME" || export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib
createdb.sh

This takes time: create database, catalog, catproc… and the you have your database

The listener is not started. We need to create the directory for the log, and to define listener.ora to listen on default port:

mkdir -p ./u01/app/oracle/product/11.2.0/xe/network/log
echo "LISTENER=(DESCRIPTION_LIST=(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = 0.0.0.0)(PORT = 1521))))" > ./u01/app/oracle/product/11.2.0/xe/network/admin/listener.ora
lsnrctl start
export ORACLE_SID=XE
sqlplus sys/oracle as sysdba <<<'alter system register;'


You should see the XE service registered here:

franck_pachot@cloudshell:~$ lsnrctl status
 
LSNRCTL for Linux: Version 11.2.0.2.0 - Production on 25-FEB-2018 23:01:40
 
Copyright (c) 1991, 2011, Oracle. All rights reserved.
 
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=0.0.0.0)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.2.0 - Production
Start Date 25-FEB-2018 23:00:01
Uptime 0 days 0 hr. 1 min. 38 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /home/frank_pachot/u01/app/oracle/product/11.2.0/xe/network/admin/listener.ora
Listener Log File /home/frank_pachot/u01/app/oracle/product/11.2.0/xe/network/log/listener.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=8080))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "XE" has 1 instance(s).
Instance "XE", status READY, has 1 handler(s) for this service...
Service "XEXDB" has 1 instance(s).
Instance "XE", status READY, has 1 handler(s) for this service...
The command completed successfully


Note that you cannot access your Google Cloud shell from outside, and then you can connect locally. But having a listener and connecting through services is always a good idea.

If your session is inactive, you may lose the connection and even have the VM stopped. But your $HOME will still be there when you restart, so you can set the .profile to set the correct environment and start the listener and database if not already running:
cat >> ~/.profile<<'END'
export ORACLE_HOME=$HOME/u01/app/oracle/product/11.2.0/xe
echo "$PATH" | grep "$ORACLE_HOME" || export PATH=$PATH:$ORACLE_HOME/bin
echo "$LD_LIBRARY_PATH" | grep "$ORACLE_HOME" || export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib
export ORACLE_SID=XE
ps -edf | grep [t]nslsnr || lsnrctl start
ps -edf | grep [s]mon_XE || sqlplus sys/oracle as sysdba <<< startup
END

I don’t use /etc/oratab here because it is outside of the persistent area.

We can not connect ‘/ as sysdba’ because we are not in the ‘dba’ group. I don’t think we can change this in Oracle XE. Of course, we can sudo to root and add the group, but that will not be persistent. However, no need for it. The password for SYS is “oracle” and you can create all the users you want. The database, being stored under $HOME, is persistent.

Here are my datafiles:

franck_pachot@cloudshell:~$ rman target sys/oracle
 
Recovery Manager: Release 11.2.0.2.0 - Production on Sun Feb 25 21:28:00 2018
 
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
 
connected to target database: XE (DBID=2850165315)
 
RMAN> report schema;
 
using target database control file instead of recovery catalog
Report of database schema for database with db_unique_name XE
 
List of Permanent Datafiles
===========================
File Size(MB) Tablespace RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1 280 SYSTEM *** /home/franck_pachot/u01/app/oracle/product/11.2.0/xe/dbs/system.dbf
2 190 SYSAUX *** /home/franck_pachot/u01/app/oracle/product/11.2.0/xe/dbs/sysaux.dbf
3 235 UNDOTBS1 *** /home/franck_pachot/u01/app/oracle/product/11.2.0/xe/dbs/undotbs1.dbf
4 100 USERS *** /home/franck_pachot/u01/app/oracle/product/11.2.0/xe/dbs/users.dbf
 
List of Temporary Files
=======================
File Size(MB) Tablespace Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1 20 TEMP 500 /home/franck_pachot/u01/app/oracle/product/11.2.0/xe/dbs/temp.dbf


You find the alert.log under $ORACLE_HOME/dbs (as all the database files):

franck_pachot@cloudshell:~$ tail $HOME/u01/app/oracle/product/11.2.0/xe/log/diag/rdbms/xe/XE/trace/alert_XE.logThread 1 advanced to log sequence 17 (LGWR switch)
Current log# 2 seq# 17 mem# 0: /home/franck_pachot/u01/app/oracle/product/11.2.0/xe/dbs/log2.dbf
Sun Feb 25 22:01:05 2018
Shared IO Pool defaulting to 44MB. Trying to get it from Buffer Cache for process 2875.
Sun Feb 25 22:09:38 2018
Thread 1 advanced to log sequence 18 (LGWR switch)
Current log# 3 seq# 18 mem# 0: /home/franck_pachot/u01/app/oracle/product/11.2.0/xe/dbs/log3.dbf
Sun Feb 25 22:09:43 2018
SERVER COMPONENT id=UTLRP_BGN: timestamp=2018-02-25 22:09:43
SERVER COMPONENT id=UTLRP_END: timestamp=2018-02-25 22:09:50

The limitations and features of the Google Cloud Shell are documented here: https://cloud.google.com/shell/docs/features. In addition to the command line (through ‘tmux’ which allows to split the screen in different panes) you have a file editor in the browser. You can also install Apex as you have browser access to port 8080 in https (icon on top right just before the menu).

The major limitation here comes from Oracle XE which is an old version (11.2.0.2) but this year should come Oracle XE 18c with the latest features. Oracle XE 18c may also come with EM Express and Google Cloud Shell gives access to https. I just hope that there will be a small image for Oracle XE 18c as we have only 5GB here. Maybe a docker container will be easier then, with only the database in an external volume under $HOME. We will see, but in the meanwhile, there’s already a lot we can do with Oracle XE. You can play with Backup/Recovery scenarios and you will always be able to re-create the database by running the createdb.sh again.

Added 26-FEB-2018

As I said that this can be a good lab to practice backup/recovery scenarios, you should run in archive log mode:

sqlplus sys/oracle as sysdba <<END
shutdown immediate;
startup mount;
alter database archivelog;
alter database open;
END

Then to avoid to fill-in the recovery area, you can backup the database and archived logs frequently. You don’t need to put the backups on the persistent storage as it is a lab.
I suggest to put the following at the end of the .profile:

rman target sys/oracle > last_backup.log <<<"set echo on; configure channel device type disk format '/var/tmp/rman_backupset_XE_%U'; configure backup optimization on; configure controlfile autobackup on; crosscheck backup; delete noprompt expired backup; backup database plus archivelog; delete noprompt obsolete;" &

This will run a backup to /var/tmp when you connect, delete obsolete backups, and expired ones (as they will be removed if the machine is reset after long inactivity).

Cet article A free persistent Google Cloud service with Oracle XE est apparu en premier sur Blog dbi services.

Remember the times when the Oracle Software features were the same on all platforms? Where Oracle databases could be ported to any relevant platform? Where we were able to try any feature, freely, by downloading the latest release software? Now we need to think differently. Because:

• The new software is released on Cloud first
• The major new features will never be available on-premises
• The Cloud here means the Oracle Cloud – not AWS, not Azure, not Google, not you local IaaS providers
• Some new features are extended to on-premises for Oracle hardware only (Exadata, ODA)
• All trial environments are paid services (but you can get free credits) but this may change with Oracle XE 18c

And you are concerned because if you start your new developments on the Oracle Cloud, or simply train yourself on new features, you may rely on features that you will never have on-premises. In my opinion, it makes sense for a startup, or a new project, to start development and early production on the Cloud. However, there will probably be a point where the cost optimization will involve on-premises servers, or IaaS, or different Cloud providers. Then the features you used may not be available.

Another concern is financial: when justifying to your CFO the cost of the 22% Support and Software Updates, you may list all the new features. But be careful. Most of the new features comes with additional options, or will not be available outside of the Oracle Cloud PaaS.

If you tried the Oracle 18c release on the Oracle Cloud, you may have seen some additional informations in the alert.log:

Capability Type : Network
capabilities requested : 1 detected : 0 Simulated : 0
Capability Type : Runtime Environment
capabilities requested : 400000FF detected : 4 Simulated : 0
Capability Type : Engineered Systems
capabilities requested : 3 detected : 0 Simulated : 0

So, it seems that Oracle is checking the capabilities of the platform to enable or not some features. When you are not on the right one, you may encounter this kind of error which is new in 18c:

[oracle@DBaaS18c ~]$ oerr ORA 12754
12754, 00000, "Feature %s is disabled due to missing capability %s."
// *Document: NO
// *Cause: This feature requires specific capabilities that were not present in the current database instance.
// *Action: Contact Oracle Support Services to enable this feature.

or maybe:

12755, 00000, "Feature %s is disabled due to unsupported capability."
// *Document: NO
// *Cause: This feature requires specific capabilities that were not supported
// in the current database instance.
// *Action: Contact Oracle Support Services to enable this feature.


For the moment, the on-premises binaries are available for Exadata only. But the Licensing documentation already gives you an idea. The following new features will not be available on-premises.

All the multitenant new features are for Oracle Cloud or Oracle Engineered systems only:

• CDB Fleet Management
• PDB Snapshot Carousel
• Refreshable PDB switchover
• Keystore for Each Pluggable Database

If you are on your own servers, or on one of the major cloud providers, you do not benefit from the latest software updates. Even if you pay each year 22% of your licenses cost, even on platforms where the core factor is maximum. You have support, and patches, but only a limited set of new features.

If you do not have the Active Data Guard option, you cannot benefit from most of the new features of the last releases. And buying this option can be expensive if you are on ULA (because you will buy it for all processors), or on non-Oracle Cloud (because of the core factor) and even there some features will not be available. The latest, Oracle Data Guard—Automatic Correction of Non-logged Blocks, is available on Oracle Cloud only, or Exadata/ODA. It is not a big problem as you can include this recovery after your nologging load, but it is important to know it.

Note that with this new release, some features also disappear. Not only deprecated. Not only desupported. But also removed. Oracle Change Data Capture has been desupported in 12c and if you look at 18c you will see that it has been removed. And it is the last version with Oracle Streams. As mentioned in the documentation, you need to buy Golden Gate.

This looks like bad news in 18c, but consider it as a new patchset on 12cR2. Remember that 12cR2 brought amazing features to all platforms and all editions, such as the online clone or move of pluggable databases. The important thing is to be informed and think differently as we used to when Oracle Databases were portable to all platforms. Be careful with features that will not be available on all platforms. Consider the costs correctly. And also look at all those features that are available to everybody and are probably not used enough. The best way is to design the application to use the database efficiently (processing data in the database, lowering the roundtrips and context switches) on a limited number of CPU cores. Then, all those options or cloud credits will not be as expensive as you may think. Nobody likes vendor lock-in, but it may be the most cost-efficient solution.

Cet article 18c, Cloud First and Cloud Only features: think differently est apparu en premier sur Blog dbi services.

SQL Plan Baselines is a great feature for plan stability: you capture the plans that you accept. However, if the data model changes and the accepted plans cannot reproduce, the optimizer will come with a new plan. In 18c we have a note from DBMS_XPLAN when the optimization ‘failed to use SQL plan baseline for this statement’.

I create a table, with an index, and run a query on it using this index:

SQL> create table DEMO as select rownum n from xmltable('1 to 1000');
Table DEMO created.
 
SQL> create index DEMO_N on DEMO(n);
Index DEMO_N created.
 
SQL> select * from DEMO where n=1;
 
N
-
1


The execution plan is correct, using the index:

SQL> select * from dbms_xplan.display_cursor();
 
PLAN_TABLE_OUTPUT
-----------------
SQL_ID 4mcr18aqntpkq, child number 0
-------------------------------------
select * from DEMO where n=1
 
Plan hash value: 217077817
 
---------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 1 (100)| |
|* 1 | INDEX RANGE SCAN| DEMO_N | 1 | 4 | 1 (0)| 00:00:01 |
---------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
1 - access("N"=1)


I’m happy with this plan, and I capture it as an accepted SQL Plan Baseline for this statemement:

SQL> exec dbms_output.put_line( dbms_spm.load_plans_from_cursor_cache('4mcr18aqntpkq') )
PL/SQL procedure successfully completed.
 
SQL> select * from dba_sql_plan_baselines;
 
SIGNATURE SQL_HANDLE SQL_TEXT PLAN_NAME CREATOR ORIGIN PARSING_SCHEMA_NAME DESCRIPTION VERSION CREATED LAST_MODIFIED LAST_EXECUTED LAST_VERIFIED ENABLED ACCEPTED FIXED REPRODUCED AUTOPURGE ADAPTIVE OPTIMIZER_COST MODULE ACTION EXECUTIONS ELAPSED_TIME CPU_TIME BUFFER_GETS DISK_READS DIRECT_WRITES ROWS_PROCESSED FETCHES END_OF_FETCH_COUNT
--------- ---------- -------- --------- ------- ------ ------------------- ----------- ------- ------- ------------- ------------- ------------- ------- -------- ----- ---------- --------- -------- -------------- ------ ------ ---------- ------------ -------- ----------- ---------- ------------- -------------- ------- ------------------
5689790730784434204 SQL_4ef632861ab7681c select * from DEMO where n=1 SQL_PLAN_4xxjkhsdbfu0wd5d62705 DEMO MANUAL-LOAD-FROM-CURSOR-CACHE DEMO 18.0.0.0.0 27-FEB-18 09.37.55.000000000 PM 27-FEB-18 09.37.55.000000000 PM YES YES NO YES YES NO 1 java@VM181 (TNS V1-V3) 3 5771 4824 82 1 0 3 6


When I run the sattement again, this plan is used and DBMS_XPLAN mentions the SQL Plan BAseline that was used:

SQL> select * from DEMO where n=1;
 
N
-
1
 
SQL> select * from dbms_xplan.display_cursor();
 
PLAN_TABLE_OUTPUT
-----------------
SQL_ID 4mcr18aqntpkq, child number 0
-------------------------------------
select * from DEMO where n=1
 
Plan hash value: 217077817
 
---------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 1 (100)| |
|* 1 | INDEX RANGE SCAN| DEMO_N | 1 | 4 | 1 (0)| 00:00:01 |
---------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
1 - access("N"=1)
 
Note
-----
- SQL plan baseline SQL_PLAN_4xxjkhsdbfu0wd5d62705 used for this statement


Now, if I drop the index, the accepted plan cannot be used:

SQL> drop index DEMO_N;
Index DEMO_N dropped.
 
SQL> select * from DEMO where n=1;
 
N
-
1
 
SQL> select * from dbms_xplan.display_cursor();
 
PLAN_TABLE_OUTPUT
-----------------
SQL_ID 4mcr18aqntpkq, child number 0
-------------------------------------
select * from DEMO where n=1
 
Plan hash value: 4000794843
 
--------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
--------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 2 (100)| |
|* 1 | TABLE ACCESS FULL| DEMO | 1 | 4 | 2 (0)| 00:00:01 |
--------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
1 - filter("N"=1)
 
Note
-----
- Failed to use SQL plan baseline for this statement


So the new note in 18c explains that there is an SQL Plan Baseline that cannot be used. Unfortunately, there is no identification of the SQL Plan baselines.


SQL> select * from dba_sql_plan_baselines;
 
SIGNATURE SQL_HANDLE SQL_TEXT PLAN_NAME CREATOR ORIGIN PARSING_SCHEMA_NAME DESCRIPTION VERSION CREATED LAST_MODIFIED LAST_EXECUTED LAST_VERIFIED ENABLED ACCEPTED FIXED REPRODUCED AUTOPURGE ADAPTIVE OPTIMIZER_COST MODULE ACTION EXECUTIONS ELAPSED_TIME CPU_TIME BUFFER_GETS DISK_READS DIRECT_WRITES ROWS_PROCESSED FETCHES END_OF_FETCH_COUNT
--------- ---------- -------- --------- ------- ------ ------------------- ----------- ------- ------- ------------- ------------- ------------- ------- -------- ----- ---------- --------- -------- -------------- ------ ------ ---------- ------------ -------- ----------- ---------- ------------- -------------- ------- ------------------
5689790730784434204 SQL_4ef632861ab7681c select * from DEMO where n=1 SQL_PLAN_4xxjkhsdbfu0w838f84a8 DEMO AUTO-CAPTURE DEMO 18.0.0.0.0 27-FEB-18 09.37.56.000000000 PM 27-FEB-18 09.37.56.000000000 PM YES NO NO YES YES NO 2 java@VM181 (TNS V1-V3) 0 0 0 0 0 0 0 0
5689790730784434204 SQL_4ef632861ab7681c select * from DEMO where n=1 SQL_PLAN_4xxjkhsdbfu0wd5d62705 DEMO MANUAL-LOAD-FROM-CURSOR-CACHE DEMO 18.0.0.0.0 27-FEB-18 09.37.55.000000000 PM 27-FEB-18 09.37.55.000000000 PM 27-FEB-18 09.37.55.000000000 PM YES YES NO YES YES NO 1 java@VM181 (TNS V1-V3) 3 5771 4824 82 1 0 3 6


So, because the accepted plan was not able to reproduce, because the index has been dropped, the new plan was captured but not accepted.

Note that if I re-create the index but with a different name, then the accepted SQL Plan cannot be used either:

SQL> create index DEMO_XXX on DEMO(n);
Index DEMO_XXX created.
 
SQL> select * from DEMO where n=1;
 
N
-
1
 
SQL> select * from dbms_xplan.display_cursor();
 
PLAN_TABLE_OUTPUT
-----------------
SQL_ID 4mcr18aqntpkq, child number 0
-------------------------------------
select * from DEMO where n=1
 
Plan hash value: 1306684165
 
-----------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-----------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 1 (100)| |
|* 1 | INDEX RANGE SCAN| DEMO_XXX | 1 | 4 | 1 (0)| 00:00:01 |
-----------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
1 - access("N"=1)
 
Note
-----
- Failed to use SQL plan baseline for this statement


So, it is not a bad idea to monitor the SQL PLan Baseline which did not reproduce. We can get them from the ‘baseline_repro_fail’ mention in OTHER_XML:

SQL> select sql_id,other_xml from v$sql_plan where other_xml like '%baseline_repro_fail%';
 
SQL_ID OTHER_XML
------ ---------
4mcr18aqntpkq <other_xml><info type="db_version">18.0.0.0</info><info type="parse_schema"><![CDATA["DEMO"]]></info><info type="plan_hash_full">211349514</info><info type="plan_hash">1306684165</info><info type="plan_hash_2">211349514</info><info type="baseline_repro_fail" note="y">yes</info><outline_data><hint><![CDATA[IGNORE_OPTIM_EMBEDDED_HINTS]]></hint><hint><![CDATA[OPTIMIZER_FEATURES_ENABLE('18.1.0')]]></hint><hint><![CDATA[DB_VERSION('18.1.0')]]></hint><hint><![CDATA[ALL_ROWS]]></hint><hint><![CDATA[OUTLINE_LEAF(@"SEL$1")]]></hint><hint><![CDATA[INDEX(@"SEL$1" "DEMO"@"SEL$1" ("DEMO"."N"))]]></hint></outline_data></other_xml>


From the SQL_ID I can get the SIGNATURE used by SQL Plan Management:

SQL> select sql_id,exact_matching_signature from v$sql where sql_id='4mcr18aqntpkq';
 
SQL_ID EXACT_MATCHING_SIGNATURE
------ ------------------------
4mcr18aqntpkq 5689790730784434204


And then the SQL Plan Baselines:

SQL> select * from dba_sql_plan_baselines where signature=5689790730784434204;
 
SIGNATURE SQL_HANDLE SQL_TEXT PLAN_NAME CREATOR ORIGIN PARSING_SCHEMA_NAME DESCRIPTION VERSION CREATED LAST_MODIFIED LAST_EXECUTED LAST_VERIFIED ENABLED ACCEPTED FIXED REPRODUCED AUTOPURGE ADAPTIVE OPTIMIZER_COST MODULE ACTION EXECUTIONS ELAPSED_TIME CPU_TIME BUFFER_GETS DISK_READS DIRECT_WRITES ROWS_PROCESSED FETCHES END_OF_FETCH_COUNT
--------- ---------- -------- --------- ------- ------ ------------------- ----------- ------- ------- ------------- ------------- ------------- ------- -------- ----- ---------- --------- -------- -------------- ------ ------ ---------- ------------ -------- ----------- ---------- ------------- -------------- ------- ------------------
5689790730784434204 SQL_4ef632861ab7681c select * from DEMO where n=1 SQL_PLAN_4xxjkhsdbfu0w0c98f00a DEMO AUTO-CAPTURE DEMO 18.0.0.0.0 27-FEB-18 10.02.07.000000000 PM 27-FEB-18 10.02.07.000000000 PM YES NO NO YES YES NO 1 java@VM181 (TNS V1-V3) 0 0 0 0 0 0 0 0
5689790730784434204 SQL_4ef632861ab7681c select * from DEMO where n=1 SQL_PLAN_4xxjkhsdbfu0w838f84a8 DEMO AUTO-CAPTURE DEMO 18.0.0.0.0 27-FEB-18 10.02.07.000000000 PM 27-FEB-18 10.02.07.000000000 PM YES NO NO YES YES NO 2 java@VM181 (TNS V1-V3) 0 0 0 0 0 0 0 0
5689790730784434204 SQL_4ef632861ab7681c select * from DEMO where n=1 SQL_PLAN_4xxjkhsdbfu0wd5d62705 DEMO MANUAL-LOAD-FROM-CURSOR-CACHE DEMO 18.0.0.0.0 27-FEB-18 10.02.06.000000000 PM 27-FEB-18 10.02.06.000000000 PM 27-FEB-18 10.02.06.000000000 PM YES YES NO YES YES NO 1 java@VM181 (TNS V1-V3) 3 4634 4210 75 1 0 3 6


Now it is easy to look at the SPM baselines to understand why it did not reproduce:

SQL> select * from dbms_xplan.display_sql_plan_baseline('SQL_4ef632861ab7681c',format=>'basic');
 
PLAN_TABLE_OUTPUT
-----------------
 
--------------------------------------------------------------------------------
SQL handle: SQL_4ef632861ab7681c
SQL text: select * from DEMO where n=1
--------------------------------------------------------------------------------
 
--------------------------------------------------------------------------------
Plan name: SQL_PLAN_4xxjkhsdbfu0w0c98f00a Plan id: 211349514
Enabled: YES Fixed: NO Accepted: NO Origin: AUTO-CAPTURE
Plan rows: From dictionary
--------------------------------------------------------------------------------
 
Plan hash value: 1306684165
 
-------------------------------------
| Id | Operation | Name |
-------------------------------------
| 0 | SELECT STATEMENT | |
| 1 | INDEX RANGE SCAN| DEMO_XXX |
-------------------------------------
 
--------------------------------------------------------------------------------
Plan name: SQL_PLAN_4xxjkhsdbfu0w838f84a8 Plan id: 2207220904
Enabled: YES Fixed: NO Accepted: NO Origin: AUTO-CAPTURE
Plan rows: From dictionary
--------------------------------------------------------------------------------
 
Plan hash value: 4000794843
 
----------------------------------
| Id | Operation | Name |
----------------------------------
| 0 | SELECT STATEMENT | |
| 1 | TABLE ACCESS FULL| DEMO |
----------------------------------
 
--------------------------------------------------------------------------------
Plan name: SQL_PLAN_4xxjkhsdbfu0wd5d62705 Plan id: 3587581701
Enabled: YES Fixed: NO Accepted: YES Origin: MANUAL-LOAD-FROM-CURSOR-CACHE
Plan rows: From dictionary
--------------------------------------------------------------------------------
 
Plan hash value: 217077817
 
-----------------------------------
| Id | Operation | Name |
-----------------------------------
| 0 | SELECT STATEMENT | |
| 1 | INDEX RANGE SCAN| DEMO_N |
-----------------------------------
 

Now we can see the unaccepted plan using index DEMO_XXX and the accepted one using index DEMO_N. If we add format=>’+outline’ we can even see that they index the same column. REady to accept the new plan and remove the old one.

In conclusion: check you V$SQL_PLAN.OTHER_XML for info type=”baseline_repro_fail” in 18c and you can do some housekeeping on SQL Plan baselines which do not reproduce. Because if you accepted a plan which cannot reproduce, you may have a problem, and better address this pro-actively.

Cet article 18c dbms_xplan note about failed SQL Plan Baseline est apparu en premier sur Blog dbi services.

There are many layers between the Oracle Database pwrite() calls and the physical sector written on disk: filesystem, logical volume, SAN or NAS, with a lot of smart software running for Virtualisation, Compression, Snapshotting, Synchronisation… Are you sure that the changes you made to your data is actually persisted on disk, completely and without any corruption? In case of bug or crash in the storage layer, it may happen that only part of the changes was written. In the case of crash, Oracle ensures that the datafile headers are written at the end, so that recovery can kick-in after the crash. Then, a partially written block can be detected and restored. With different checksum settings, you can also check block integrity while writing or reading. But that protects only for fractured blocks. What if a block write just did not occur? An old version of the block remains and then is perfectly correct for checksum, RMAN, and DBV.

You may be 100% sure that you have never experienced lost writes. But then I’ll ask you: how do you know it? You don’t. Except if you enable Lost Write Protection.

In 11g Oracle introduced this feature for Data Guard configurations. Data Guard is the best protection as the synchronization is done at the highest level: the change vector, generated before any I/O and block modification occurred. Do not use SAN synchronization for your database. Data Guard is less expensive (no option needed), more efficient (only the persistent change information is shipped), and protects over all layers. It protects from lost writes because blocks are written on both sites by a different server, instance, storage. And Data Guard can detect lost writes by shipping the block SCN for each read to compare it with the standby.

However, this has an overhead: redo generation for reads. Oracle 18c comes with a new solution with no need for a standby database: a new LOST WRITE PROTECTION tablespace is created to store the SCN of each block modified.

Do you have lost writes?

First I’ll show what happens without this feature. I create a table filled with number, timestamp and SCN:

SQL> create table DEMO.DEMO pctfree 99 as select rownum id,1 n, current_timestamp ts , (select current_scn from v$database) scn from xmltable('1 to 10');
Table DEMO.DEMO created.
 
SQL> select owner,segment_name,segment_type,block_id,blocks,sum(blocks)over(partition by owner,segment_name,segment_type) from dba_extents where owner='DEMO' and segment_name='DEMO' order by 1,2,3,4;
 
OWNER SEGMENT_NAME SEGMENT_TYPE BLOCK_ID BLOCKS SUM(BLOCKS)OVER(PARTITIONBYOWNER,SEGMENT_NAME,SEGMENT_TYPE)
----- ------------ ------------ -------- ------ -----------------------------------------------------------
DEMO DEMO TABLE 3128 8 8
 
SQL> column block_id new_value last_block_id
SQL> select dbms_rowid.rowid_block_number(rowid) block_id,DEMO.* from DEMO.DEMO;
 
BLOCK_ID ID N TS SCN
-------- -- - -- ---
3131 1 1 03-MAR-18 04.51.37.838991000 PM EUROPE/ZURICH 4619734
3131 2 1 03-MAR-18 04.51.37.838991000 PM EUROPE/ZURICH 4619734
3132 3 1 03-MAR-18 04.51.37.838991000 PM EUROPE/ZURICH 4619734
3132 4 1 03-MAR-18 04.51.37.838991000 PM EUROPE/ZURICH 4619734
3133 5 1 03-MAR-18 04.51.37.838991000 PM EUROPE/ZURICH 4619734
3133 6 1 03-MAR-18 04.51.37.838991000 PM EUROPE/ZURICH 4619734
3134 7 1 03-MAR-18 04.51.37.838991000 PM EUROPE/ZURICH 4619734
3134 8 1 03-MAR-18 04.51.37.838991000 PM EUROPE/ZURICH 4619734
3135 9 1 03-MAR-18 04.51.37.838991000 PM EUROPE/ZURICH 4619734
3135 10 1 03-MAR-18 04.51.37.838991000 PM EUROPE/ZURICH 4619734
 
SQL> column block_id clear


I’ve recorded the las BLOCK_ID in &last_block_id and ensures that all those modifications are written on dosk:

SQL> alter system checkpoint;
System CHECKPOINT altered.
 
SQL> alter system flush buffer_cache;
System FLUSH altered.


I save the block 3133 to keep an old version of it. This will be my way to simulate a lost write.

SQL> host dd if=/u01/oradata/CDB1/PDB1/users01.dbf of=/var/tmp/lwp.blk skip=$(( &last_block_id - 2 )) bs=8k count=1
1+0 records in
1+0 records out
8192 bytes (8.2 kB) copied, 0.000263416 s, 31.1 MB/s


Now, I update all rows, set new timestamp, SCN and increase the number 1 to 2.

SQL> update DEMO.DEMO set n=2+1, ts=current_timestamp, scn=(select current_scn from v$database);
10 rows updated.
SQL> commit;
Commit complete.
 
SQL> alter system checkpoint;
System CHECKPOINT altered.
SQL> alter system flush buffer_cache;
System FLUSH altered.


Here is how I reproduce lost writes. All blocks changed were written to disk, but I restore the old version of block 3133 as if this one was not written:

SQL> host dd if=/var/tmp/lwp.blk of=/u01/oradata/CDB1/PDB1/users01.dbf seek=$(( &last_block_id - 2 )) bs=8k count=1 conv=notrunc
1+0 records in
1+0 records out
8192 bytes (8.2 kB) copied, 0.000111582 s, 73.4 MB/s


This is what you can see if one of your storage layers missed a write and nevertheless acknowledged the I/O call.
s
SQL> select dbms_rowid.rowid_block_number(rowid) block_id,DEMO.* from DEMO.DEMO;
BLOCK_ID ID N TS SCN
-------- -- - --------------------------------------------- -------
3131 1 3 03-MAR-18 04.51.39.255582000 PM EUROPE/ZURICH 4619806
3131 2 3 03-MAR-18 04.51.39.255582000 PM EUROPE/ZURICH 4619806
3132 3 3 03-MAR-18 04.51.39.255582000 PM EUROPE/ZURICH 4619806
3132 4 3 03-MAR-18 04.51.39.255582000 PM EUROPE/ZURICH 4619806
3133 5 1 03-MAR-18 04.51.37.838991000 PM EUROPE/ZURICH 4619734
3133 6 1 03-MAR-18 04.51.37.838991000 PM EUROPE/ZURICH 4619734
3134 7 3 03-MAR-18 04.51.39.255582000 PM EUROPE/ZURICH 4619806
3134 8 3 03-MAR-18 04.51.39.255582000 PM EUROPE/ZURICH 4619806
3135 9 3 03-MAR-18 04.51.39.255582000 PM EUROPE/ZURICH 4619806
3135 10 3 03-MAR-18 04.51.39.255582000 PM EUROPE/ZURICH 4619806


No errors. No corruption. Just old values for the rows in this block. And there’s no way to detect it. Physical reads, RMAN, DBV, will all see a perfectly correct block. Only if those changes are inconsistent with other tables (with constraints) or indexes, you may detect a logical corruption.

The probability that this problem occurs is very low (a storage bug on exactly one 8k block or multiple of it). But it is critical because it cannot be detected. To detect it, you need to compare the full block or a checksum, or simply the SCN with another copy, such as in the standby database. Or, with this new feature, store the SCN of each data block in a new structure: the 18c LOST WRITE DETECTION.

Enabling Lost Write Protection

You need to create a tablespace to store those SCN. There’s no choice you must use a bigfile tablespace, but you can create multiple small ones if you want.

SQL> create bigfile tablespace SHADOW
2 datafile '/u01/oradata/CDB1/PDB1/shadow.dbf'
3 size 5M
4 lost write protection;
 
Tablespace SHADOW created.
 
SQL> select tablespace_name,status,bigfile,contents,logging,allocation_type,encrypted,lost_write_protect,chunk_tablespace from dba_tablespaces;
TABLESPACE_NAME STATUS BIGFILE CONTENTS LOGGING ALLOCATION_TYPE ENCRYPTED LOST_WRITE_PROTECT CHUNK_TABLESPACE
--------------- ------ ------- -------- ------- --------------- --------- ------------------ ----------------
SYSTEM ONLINE NO PERMANENT LOGGING SYSTEM NO OFF N
SYSAUX ONLINE NO PERMANENT LOGGING SYSTEM NO OFF N
UNDOTBS1 ONLINE NO UNDO LOGGING SYSTEM NO OFF N
TEMP ONLINE NO TEMPORARY NOLOGGING UNIFORM NO OFF N
SHADOW ONLINE YES LOST WRITE PROTECTION LOGGING SYSTEM NO OFF N
USERS ONLINE NO PERMANENT LOGGING SYSTEM NO OFF N
 
SQL> select * from dba_data_files;
 
FILE_NAME FILE_ID TABLESPACE_NAME BYTES BLOCKS STATUS RELATIVE_FNO AUTOEXTENSIBLE MAXBYTES MAXBLOCKS INCREMENT_BY USER_BYTES USER_BLOCKS ONLINE_STATUS LOST_WRITE_PROTECT
--------- ------- --------------- ----- ------ ------ ------------ -------------- -------- --------- ------------ ---------- ----------- ------------- ------------------
/u01/oradata/CDB1/PDB1/undotbs01.dbf 164 UNDOTBS1 104857600 12800 AVAILABLE 9 YES 34359721984 4194302 640 103809024 12672 ONLINE OFF
/u01/oradata/CDB1/PDB1/sysaux01.dbf 163 SYSAUX 408944640 49920 AVAILABLE 4 YES 34359721984 4194302 1280 407896064 49792 ONLINE OFF
/u01/oradata/CDB1/PDB1/system01.dbf 162 SYSTEM 272629760 33280 AVAILABLE 1 YES 34359721984 4194302 1280 271581184 33152 SYSTEM OFF
/u01/oradata/CDB1/PDB1/users01.dbf 169 USERS 104857600 12800 AVAILABLE 169 NO 0 0 0 103809024 12672 ONLINE OFF
/u01/oradata/CDB1/PDB1/shadow.dbf 58 SHADOW 5242880 640 AVAILABLE 1024 NO 0 0 0 4194304 512 ONLINE OFF


Then you enable this feature for the database (or pluggable database) and for the tablespaces you want to protect:

SQL> alter pluggable database enable lost write protection;
Pluggable database ENABLE altered.
 
SQL> alter tablespace USERS enable lost write protection;
Tablespace USERS altered.


Here are the new columns in DBA_TABLESPACES and DBA_DATA_FILES:

SQL> select tablespace_name,status,bigfile,contents,logging,allocation_type,encrypted,lost_write_protect,chunk_tablespace from dba_tablespaces;
 
TABLESPACE_NAME STATUS BIGFILE CONTENTS LOGGING ALLOCATION_TYPE ENCRYPTED LOST_WRITE_PROTECT CHUNK_TABLESPACE
--------------- ------ ------- -------- ------- --------------- --------- ------------------ ----------------
SYSTEM ONLINE NO PERMANENT LOGGING SYSTEM NO OFF N
SYSAUX ONLINE NO PERMANENT LOGGING SYSTEM NO OFF N
UNDOTBS1 ONLINE NO UNDO LOGGING SYSTEM NO OFF N
TEMP ONLINE NO TEMPORARY NOLOGGING UNIFORM NO OFF N
SHADOW ONLINE YES LOST WRITE PROTECTION LOGGING SYSTEM NO OFF N
USERS ONLINE NO PERMANENT LOGGING SYSTEM NO ENABLED N
 
SQL> select * from dba_data_files;
 
FILE_NAME FILE_ID TABLESPACE_NAME BYTES BLOCKS STATUS RELATIVE_FNO AUTOEXTENSIBLE MAXBYTES MAXBLOCKS INCREMENT_BY USER_BYTES USER_BLOCKS ONLINE_STATUS LOST_WRITE_PROTECT
--------- ------- --------------- ----- ------ ------ ------------ -------------- -------- --------- ------------ ---------- ----------- ------------- ------------------
/u01/oradata/CDB1/PDB1/undotbs01.dbf 164 UNDOTBS1 104857600 12800 AVAILABLE 9 YES 34359721984 4194302 640 103809024 12672 ONLINE OFF
/u01/oradata/CDB1/PDB1/sysaux01.dbf 163 SYSAUX 408944640 49920 AVAILABLE 4 YES 34359721984 4194302 1280 407896064 49792 ONLINE OFF
/u01/oradata/CDB1/PDB1/system01.dbf 162 SYSTEM 272629760 33280 AVAILABLE 1 YES 34359721984 4194302 1280 271581184 33152 SYSTEM OFF
/u01/oradata/CDB1/PDB1/users01.dbf 169 USERS 104857600 12800 AVAILABLE 169 NO 0 0 0 103809024 12672 ONLINE ENABLED
/u01/oradata/CDB1/PDB1/shadow.dbf 58 SHADOW 5242880 640 AVAILABLE 1024 NO 0 0 0 4194304 512 ONLINE OFF


Note that we are on the Oracle Cloud here and all tablespaces must be encrypted. This is also the case with the LOST WRITE PROTECTION one. Here you can see ENCRYPTION at none only because I decrypted them to look at what is inside the files.

Here are some internal tables giving some information about the storage. Note that tablespace number TSID=7 here is USERS, the one protected, and the TSID=6 one is the LOST WRITE PROTECTION one.

SQL> select * from new_lost_write_datafiles$;
 
DATAFILE_TSID_TRACKED DATAFILE_RFN_TRACKED SHADOW_DATAFILE_TSID SHADOW_DATAFILE_RFN SHADOW_DATAFILE_OFFSET NUMBER_OF_BLOCKS_ALLOCATED DATAFILE_CURRENT_STATUS
--------------------- -------------------- -------------------- ------------------- ---------------------- -------------------------- -----------------------
7 169 6 1024 128 184 enabled
 
SQL> select * from new_lost_write_extents$;
 
EXTENT_DATAFILE_RFN EXTENT_DATAFILE_TSID EXTENT_START EXTENT_LENGTH_BLOCKS_2K EXTENT_NEXT_BLOCK
------------------- -------------------- ------------ ----------------------- -----------------
1024 6 312 1312 641
 
SQL> select * from new_lost_write_shadows$;
 
SHADOW_DATAFILE_RFN SHADOW_DATAFILE_TSID SHADOW_NUMBER_BLOCKS_ALLOC SHADOW_FIRST_FREE_BLOCK SHADOW_BLOCK_SIZE_BYTES SHADOW_RECS_PER_BLOCK
------------------- -------------------- -------------------------- ----------------------- ----------------------- ---------------------
1024 6 640 128 8192 136


Because there was already a ‘lost write’ feature, this one is called ‘new lost write’.

NEW_LOST_WRITE_DATAFILE$ lists all protected (aka tracked) datafiles with the LOST WRITE PROTECTION (aka shadow) tablespace protecting it. The status can be ‘enabled’ or ‘suspended’. The row is deleted if the protection is removed.

NEW_LOST_WRITE_SHADOWS$ lists the LOST WRITE PROTECTION tablespaces. It contains a 1MB bitmap in the first 128 blocks, and extents starts after this.

NEW_LOST_WRITE_EXTENTS$ maps the free extents in the shadow tablespace. Here, the lost write protection for my USERS tablespace (100MB) takes 312 – 128 = 184 blocks (1.4 MB). The extent is 4MB. The 1312 EXTENT_LENGTH_BLOCKS_2K are the remaining free space in the extent, in 2KB blocks. 4MB-1.4MB=2.6MB which is 1312 x 2k blocks.

Simulate Lost Write

As I did before, I create a DEMO table

SQL> create table DEMO.DEMO pctfree 99 as select rownum id,1 n, current_timestamp ts , (select current_scn from v$database) scn from xmltable('1 to 10');
Table DEMO.DEMO created.
 
SQL> column block_id new_value last_block_id
SQL> select dbms_rowid.rowid_block_number(rowid) block_id,DEMO.* from DEMO.DEMO;
BLOCK_ID ID N TS SCN
-------- -- - --------------------------------------------- -------
3123 1 1 03-MAR-18 04.21.00.797975000 PM EUROPE/ZURICH 4578079
3123 2 1 03-MAR-18 04.21.00.797975000 PM EUROPE/ZURICH 4578079
3124 3 1 03-MAR-18 04.21.00.797975000 PM EUROPE/ZURICH 4578079
3124 4 1 03-MAR-18 04.21.00.797975000 PM EUROPE/ZURICH 4578079
3125 5 1 03-MAR-18 04.21.00.797975000 PM EUROPE/ZURICH 4578079
3125 6 1 03-MAR-18 04.21.00.797975000 PM EUROPE/ZURICH 4578079
3126 7 1 03-MAR-18 04.21.00.797975000 PM EUROPE/ZURICH 4578079
3126 8 1 03-MAR-18 04.21.00.797975000 PM EUROPE/ZURICH 4578079
3127 9 1 03-MAR-18 04.21.00.797975000 PM EUROPE/ZURICH 4578079
3127 10 1 03-MAR-18 04.21.00.797975000 PM EUROPE/ZURICH 4578079

I save one block:

SQL> column block_id clear
SQL> alter system checkpoint;
System CHECKPOINT altered.
SQL> alter system flush buffer_cache;
System FLUSH altered.
SQL> host dd if=/u01/oradata/CDB1/PDB1/users01.dbf of=/var/tmp/lwp.blk skip=$(( &last_block_id - 2 )) bs=8k count=1
1+0 records in
1+0 records out
8192 bytes (8.2 kB) copied, 0.000325309 s, 25.2 MB/s

Update the table:

SQL> update DEMO.DEMO set n=2+1, ts=current_timestamp, scn=(select current_scn from v$database);
10 rows updated.
SQL> commit;
Commit complete.
SQL> alter system checkpoint;
System CHECKPOINT altered.
SQL> alter system flush buffer_cache;
System FLUSH altered.

Write back the block I saved, to simulate a lost write:

SQL> host dd if=/var/tmp/lwp.blk of=/u01/oradata/CDB1/PDB1/users01.dbf seek=$(( &last_block_id - 2 )) bs=8k count=1 conv=notrunc
1+0 records in
1+0 records out
8192 bytes (8.2 kB) copied, 0.000104103 s, 78.7 MB/s


Lost Write detection

Now, when I query the table, the lost write is detected and an error is raised:

SQL> select dbms_rowid.rowid_block_number(rowid) block_id,DEMO.* from DEMO.DEMO;
 
Error starting at line : 93 File @ /media/sf_share/18c/lost_write_protection.sql
In command -
select dbms_rowid.rowid_block_number(rowid) block_id,DEMO.* from DEMO.DEMO
Error report -
ORA-65478: shadow lost write protection - found lost write


In the alert.log I have the mention of the block that failed:

2018-03-03 16:21:06.842000 +01:00
ERROR - I/O type:buffered I/O found lost write in block with file#:169 rdba:0x2a400c35, Expected SCN:0x000000000045db54 SCN in block:0x000000000045db23, approx current SCN:0x000000000045dbbb, RAC instance:1 pdb:5
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************


The block defined by the RDBA in alert.log is the one I have manually corrupted:

SQL> select dbms_utility.data_block_address_file(to_number('2a400c35','XXXXXXXX'))file#,dbms_utility.data_block_address_block( to_number('2a400c35','XXXXXXXX'))block# from dual;
 
FILE# BLOCK#
---------- ----------
169 3125


As mentioned in the alert.log the session has dumped the redo, as found in the session trace file:

ALTER SYSTEM DUMP REDO DBA MIN 169 3125 DBA MAX 169 3125 SCN MIN 4578132 SCN MAX 4578235 CON_ID 5

This SCN 4578132 is the commit SCN for my update. And the 4578235 is the current one. I can see the change that was lost here:

CHANGE #10 CON_ID:5 TYP:0 CLS:1 AFN:169 DBA:0x2a400c35 OBJ:73527 SCN:0x000000000045db23 SEQ:2 OP:11.4 ENC:0 RBL:0 FLG:0x0000
KTB Redo
op: 0x01 ver: 0x01
compat bit: 4 (post-11) padding: 1
op: F xid: 0x0008.019.000002d6 uba: 0x02401257.01a2.24
KDO Op code: LKR row dependencies Disabled
xtype: XA flags: 0x00000000 bdba: 0x2a400c35 hdba: 0x2a400c32
itli: 2 ispac: 0 maxfr: 4858
tabn: 0 slot: 0 to: 2
CHANGE #11 CON_ID:5 TYP:0 CLS:1 AFN:169 DBA:0x2a400c35 OBJ:73527 SCN:0x000000000045db54 SEQ:1 OP:11.5 ENC:0 RBL:0 FLG:0x0000
KTB Redo
op: 0x02 ver: 0x01
compat bit: 4 (post-11) padding: 1
op: C uba: 0x02401257.01a2.25
KDO Op code: URP row dependencies Disabled
xtype: XAxtype KDO_KDOM2 flags: 0x00000080 bdba: 0x2a400c35 hdba: 0x2a400c32
itli: 2 ispac: 0 maxfr: 4858
tabn: 0 slot: 0(0x0) flag: 0x2c lock: 2 ckix: 0
ncol: 4 nnew: 3 size: 0
Vector content:
col 1: [ 2] c1 04
col 2: [13] 78 76 03 03 10 16 02 0d 73 cd 48 86 58
col 3: [ 5] c4 05 3a 52 20


Then we need to recover…

However, unfortunately, this block is not marked as corrupt:

SQL> select * from V$DATABASE_BLOCK_CORRUPTION;
no rows selected


This means that I cannot use RMAN block recovery:

SQL> host rman target / <<
Starting recover at 03-MAR-18
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=152 device type=DISK
allocated channel: ORA_DISK_2
 
starting media recovery
media recovery complete, elapsed time: 00:00:00
 
Finished recover at 03-MAR-18


And the RMAN recovery advisor is not aware of the problem:

RMAN> list failure;
 
using target database control file instead of recovery catalog
Database Role: PRIMARY
 
no failures found that match specification


So the solution is to mark it as corrupt or restore the whole datafile (or section). And maybe ensure that the write was lost on the data, and not on the lost write tracking block itself. The redo dump may help for that.

strace: additional I/Os

I traced the server process to see what files are read during my query:

SQL> column spid new_value pid
SQL> select spid from v$process join v$session on v$session.paddr=v$process.addr where sid=sys_context('userenv','sid');
SPID
9360
SQL> column spid clear
SQL> define bg=&:
SQL> host strace -p &pid -o strace.txt &bg


This small awk filters I/O calls on users01.dbf and shadow.dbf and displays the system calls on the file handle

awk '/open[(]["].*(shadow.dbf|users01.dbf).*["],/{l=$0;gsub(/[(,)]/," ");h[$NF]=$2" "$NF;print $l;$0=$l}/^[a-zA-Z0-9]+[(][0-9]+[,)]/{l=$0;gsub(/[(,)]/," "); if ( $1 == "close" ) h[$2]="" ; if ( h[$2]!="" ) printf "%-130s \t%80s\n",l,h[$2]}/F_DUPFD/{if ( h[$2]!="" ) h[$NF]=h[$2]" "$NF;h[$2]=""}' strace.txt | grep --color=auto -E "^|^.*users01.*"


Here are the open() and pread() calls:

open "/u01/oradata/CDB1/PDB1/shadow.dbf" O_RDWR|O_DSYNC = 8
fcntl(8, F_SETFD, FD_CLOEXEC) = 0 "/u01/oradata/CDB1/PDB1/shadow.dbf" 8
fcntl(8, F_DUPFD, 256) = 256 "/u01/oradata/CDB1/PDB1/shadow.dbf" 8
fcntl(256, F_SETFD, FD_CLOEXEC) = 0 "/u01/oradata/CDB1/PDB1/shadow.dbf" 8 256
pread64(256, "X\242\226V\333E\1\f\372\366"..., 8192, 1056768) = 8192 "/u01/oradata/CDB1/PDB1/shadow.dbf" 8 256
open "/u01/oradata/CDB1/PDB1/users01.dbf" O_RDWR|O_DSYNC = 8
fcntl(8, F_SETFD, FD_CLOEXEC) = 0 "/u01/oradata/CDB1/PDB1/users01.dbf" 8
fcntl(8, F_DUPFD, 256) = 257 "/u01/oradata/CDB1/PDB1/users01.dbf" 8
fcntl(257, F_SETFD, FD_CLOEXEC) = 0 "/u01/oradata/CDB1/PDB1/users01.dbf" 8 257
pread64(257, "#\242002\f@*%\333E\1\4\370\264"..., 8192, 25575424) = 8192 "/u01/oradata/CDB1/PDB1/users01.dbf" 8 257
pread64(257, "\6\242003\f@*T\333E\5\6\317H\1007\37\1!\333E"..., 40960, 25583616) = 40960 "/u01/oradata/CDB1/PDB1/users01.dbf" 8 257
pread64(257, "\6\242005\f@*#\333E\2\4zo\1007\37\1!\333E"..., 8192, 25600000) = 8192 "/u01/oradata/CDB1/PDB1/users01.dbf" 8 257
pread64(257, "\6\242003\f@*T\333E\5\6\317H\1007\37\1!\333E"..., 40960, 25583616) = 40960 "/u01/oradata/CDB1/PDB1/users01.dbf" 8 257
pread64(257, "\6\242005\f@*#\333E\2\4zo\1007\37\1!\333E"..., 8192, 25600000) = 8192 "/u01/oradata/CDB1/PDB1/users01.dbf" 8 257
pread64(257, "\6\242003\f@*T\333E\5\6\317H\1007\37\1!\333E"..., 40960, 25583616) = 40960 "/u01/oradata/CDB1/PDB1/users01.dbf" 8 257
pread64(257, "\6\242005\f@*#\333E\2\4zo\1007\37\1!\333E"..., 8192, 25600000) = 8192 "/u01/oradata/CDB1/PDB1/users01.dbf" 8 257


We can see the lost write protection file read first (1 block at offset 1056768 which is block 129, the first one after the 1MB header) and the SCNs for my 5 blocks table are all there. Then the table blocks are read. Note that all those blocks (lost protection and data) goes into the buffer cache, and then do not have to be re-read each time. Here, I’ve run my failing select 3 times and only the first one had to read the shadow datafile.

X$BH: additional buffer gets

As those blocks are read through the buffer cache during the consistent reads, I checked the buffer cache headers for the 3 times I’ve run the queries. I’ve identified them from the function that reads them: kcbr_lost_get_lost_write_scns

SQL> select dbms_rowid.rowid_block_number(rowid) block_id,DEMO.* from DEMO.DEMO;
ORA-65478: shadow lost write protection - found lost write
 
SQL> select obj,state,tch,fp_whr from x$bh where fp_whr like 'kr_gcur_4: kcbr_lost_get_lost_w%';
 
OBJ STATE TCH FP_WHR
--- ----- --- ------
4294967295 1 1 kr_gcur_4: kcbr_lost_get_lost_w
 
 
SQL> select dbms_rowid.rowid_block_number(rowid) block_id,DEMO.* from DEMO.DEMO;
ORA-65478: shadow lost write protection - found lost write
 
SQL> select obj,state,tch,fp_whr from x$bh where fp_whr like 'kr_gcur_4: kcbr_lost_get_lost_w%';
 
OBJ STATE TCH FP_WHR
--- ----- --- ------
4294967295 1 2 kr_gcur_4: kcbr_lost_get_lost_w
 
 
SQL> select dbms_rowid.rowid_block_number(rowid) block_id,DEMO.* from DEMO.DEMO;
ORA-65478: shadow lost write protection - found lost write
 
SQL> select obj,state,tch,fp_whr from x$bh where fp_whr like 'kr_gcur_4: kcbr_lost_get_lost_w%';
 
OBJ STATE TCH FP_WHR
--- ----- --- ------
4294967295 1 3 kr_gcur_4: kcbr_lost_get_lost_w


Here we can see the touch count increasing. It seems that for each query the kcbr_lost_get_lost_write_scns is called, even when there was no modification and no new read from disk.

While we’re there, let’s breakpoint on this fonction to see when it is called:

(gdb) break kcbr_lost_get_lost_write_scns
Breakpoint 1 at 0x85a9140
(gdb) c
Continuing.
 
Breakpoint 1, 0x00000000085a9140 in kcbr_lost_get_lost_write_scns ()
(gdb) bt
#0 0x00000000085a9140 in kcbr_lost_get_lost_write_scns ()
#1 0x0000000001cf9c01 in kcbzibmlt ()
#2 0x0000000001ce2f29 in kcbzib ()
#3 0x0000000011e7f6e9 in kcbgtcr ()
#4 0x0000000011e366bd in ktrget2 ()
#5 0x00000000121d5ca7 in kdst_fetch0 ()
#6 0x00000000121e4f5a in kdstf000110100000000km ()
#7 0x00000000121d398e in kdsttgr ()
#8 0x000000001224d28f in qertbFetch ()
#9 0x00000000120340ef in opifch2 ()
#10 0x0000000002d8d033 in kpoal8 ()
#11 0x000000001203af9c in opiodr ()
#12 0x000000001230acf7 in ttcpip ()
#13 0x00000000026a5667 in opitsk ()
#14 0x00000000026aa27d in opiino ()
#15 0x000000001203af9c in opiodr ()
#16 0x00000000026a10a3 in opidrv ()
#17 0x00000000032a58af in sou2o ()
#18 0x0000000000d68047 in opimai_real ()
#19 0x00000000032b2667 in ssthrdmain ()
#20 0x0000000000d67e53 in main ()

Look at Frits Hoogland annotations for the signification and you will see that this is called during consistent reads -> input buffer.

So what?

This feature is interesting. Of course, we need to measure the overhead of this detection, but this additional storage of the SCN being implemented as any data block, benefits from all its efficiency: buffer cache, background writes by dbwr, protection by redo, backups,… These times, I see more and more databases installed on storage with fancy features, and admins playing with snapshot without really knowing whether it is consistent or not. This is the opposite of the ‘reliable’ and ‘keep it simple’ properties that we want for our data. For these environments, when I cannot convince the storage admins to forget about those features and rely on Data Guard on top of the simplest storage, then at least we have a way to protect us from failures in those layers.

Cet article 18c new Lost Write Protection est apparu en premier sur Blog dbi services.

To perform some tests in order to determine the accurate number of cores to be used by the database application, I had to increase and decrease the number of CPU cores on an ODA using the DCS stack.  When increasing would never be a problem, decreasing number of CPU cores would fail into a DCS-10045 error.

The ODA was initialized with 6 CPU cores, and purpose was to reduce it to 4 CPU cores (of course for testing ).

[root@BApdl006060 ~]# odacli list-cpucores

Node  Cores  Modified                       Job Status
----- ------ ------------------------------ ---------------
0     36     February 23, 2018 12:22:12 AM CET Configured
0     6      February 26, 2018 11:41:06 AM CET Configured

Running the below command to reduce CPU cores

[root@BApdl006060 ~]# odacli update-cpucore -c 4

Unfortunately this failed into the error :

DCS-10045:Validation error encountered: reduction in number of cores is not supported

If failing into this case, the tip would be to use the “force” option.

[root@BApdl006060 ~]# odacli update-cpucore -c 4 --force
{
  "jobId" : "5d343d01-6eb1-47f1-af2b-7df3f8adab84",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "February 28, 2018 08:49:21 AM CET",
  "resourceList" : [ ],
  "description" : "CPU cores  service update",
  "updatedTime" : "February 28, 2018 08:49:21 AM CET"
}

Once the job is completed successfully, the ODA is running the expected 4 CPU cores.

root@BApdl006060 ~]# odacli list-cpucores

Node  Cores  Modified                       Job Status
----- ------ ------------------------------ ---------------
0     36     February 23, 2018 12:22:12 AM CET Configured
0     6      February 26, 2018 11:41:06 AM CET Configured
0     4      February 28, 2018 8:49:29 AM CET Configured

[root@BApdl006060 ~]# odacli describe-cpucore

Node  Cores  Modified                       Job Status
----- ------ ------------------------------ ---------------
0     4      February 28, 2018 8:49:29 AM CET Configured

Of course, knowing that reducing the CPU cores is not supported, recommendation would be to use the force option only in test environment and not in production.

Cet article ODA – Reducing number of cores : Getting error DCS-10045 est apparu en premier sur Blog dbi services.

When the documentation is not always clear, I prefer to build a test case to be sure about the behavior in different context and different versions. Here is a test on SQL Plan Management to show which plan is chosen among the different states of SQL Plan Baselines: Enabled, Accepted, Fixed. Thanks to Oracle ACE program, I have some Oracle Cloud credits to quickly provision a database, so I tested that on Oracle 18c.

For this test, I’ve created a table:

create table DEMO as select rownum n from xmltable('1 to 10000');

with 8 indexes:

exec for i in 1..8 loop execute immediate 'create index DEMO'||i||' on DEMO(n,'||i||')'; end loop;

and a procedure to query it several times, setting random costs for the indexes, with only one cheapest:

create or replace procedure runplans(n number) as
dummy number;
begin
-- run all this 30 times
for k in 1..30 loop
run from index DEMO1 to DEMOt with one of them cheaper each time
for t in 1..n loop
-- set random cost for all indexes
for i in 1..8 loop dbms_stats.set_index_stats(user,'DEMO'||i,indlevel=>round(dbms_random.value(10,100)),no_invalidate=>true); end loop;
-- set cheap cost for index DEMOt
dbms_stats.set_index_stats(user,'DEMO'||t,indlevel=>1,no_invalidate=>true);
-- change some parameters to parse new child
execute immediate 'alter session set optimizer_index_caching='||(t*8+k);
-- query with an index hint but not specifying the index so the cheaper is chosen
select /*+ index(DEMO) */ n into dummy from DEMO where n=1;
end loop;
end loop;
end;
/

So you understand the idea: have 8 possible execution plans, with one cheaper than the others. And the goal is to see which one is chosen depending of the state of the SQL Plan Baseline.

I will play with the baselines and will display the cursor execution with the following SQLcl alias:

SQL> alias sqlpb=select sql_id,plan_hash_value,sum(executions),sql_plan_baseline,id,object_name from v$sql join v$sql_plan using(sql_id,plan_hash_value,child_number) where sql_text like 'SELECT % FROM DEMO%' and id=1 group by sql_id,plan_hash_value,sql_plan_baseline,id,object_name order by 3;


So, I call my procedure to run the query 30 times when index DEMO1 is the cheapest:

SQL> exec runplans(1)
PL/SQL procedure successfully completed.


Here is my alias to show the 30 executions using DEMO1 (object_name from v$sqlplan):

SQL> sqlpb
 
SQL_ID PLAN_HASH_VALUE SUM(EXECUTIONS) SQL_PLAN_BASELINE ID OBJECT_NAME
------------- --------------- --------------- -------------------- -- -----------
gqnkn2d17zjvv 3739632713 30 1 DEMO1


I load this to SPM

SQL> exec dbms_output.put_line(dbms_spm.load_plans_from_cursor_cache(sql_id=>'gqnkn2d17zjvv'));
PL/SQL procedure successfully completed.


Here is my SQL Plan Baseline, enabled and accepted:

SQL> select plan_name,enabled,accepted,fixed,executions from dba_sql_plan_baselines;
 
PLAN_NAME ENABLED ACCEPTED FIXED EXECUTIONS
------------------------------ ------- -------- ----- ----------
SQL_PLAN_dcc9d14j7k1vu97e16a35 YES YES NO 30


Now I run my procedure to run 30 times the cursor and for each, 8 times with one different index being the cheapest:

SQL> exec runplans(8)
PL/SQL procedure successfully completed.


So all executions have used the only one SQL Plan Baseline which is enabled and accepted:

SQL> sqlpb
 
SQL_ID PLAN_HASH_VALUE SUM(EXECUTIONS) SQL_PLAN_BASELINE ID OBJECT_NAME
------------- --------------- --------------- --------------------------------- -- -----------
gqnkn2d17zjvv 3739632713 232 SQL_PLAN_dcc9d14j7k1vu97e16a35 1 DEMO1


And the other plans (because only 8 of them had this DEMO1 plan being the cheapest) were loaded, enabled but not accepted:

SQL> select plan_name,enabled,accepted,fixed,executions from dba_sql_plan_baselines;
 
PLAN_NAME ENABLED ACCEPTED FIXED EXECUTIONS
------------------------------ ------- -------- ----- ----------
SQL_PLAN_dcc9d14j7k1vu287d1344 YES NO NO 0
SQL_PLAN_dcc9d14j7k1vu452ab005 YES NO NO 0
SQL_PLAN_dcc9d14j7k1vu4564f9cd YES NO NO 0
SQL_PLAN_dcc9d14j7k1vu4cdc9ee7 YES NO NO 0
SQL_PLAN_dcc9d14j7k1vu5353a77e YES NO NO 0
SQL_PLAN_dcc9d14j7k1vu97e16a35 YES YES NO 30
SQL_PLAN_dcc9d14j7k1vuc6a3f323 YES NO NO 0
SQL_PLAN_dcc9d14j7k1vufb8f9e5a YES NO NO 0


Now, I change the status of the baselines to get all combinations of enabled, fixed and accepted, and in addition to that change the plan name to tne line of plan which differs:

SQL> begin
2 for i in (select rownum n,plan_name from dba_sql_plan_baselines) loop
3 if i.n in (2,4,6,8) then dbms_output.put_line(dbms_spm.evolve_sql_plan_baseline(plan_name=>i.plan_name,verify=>'no',commit=>'yes')); end if;
4 if i.n in (1,2,5,6) then dbms_output.put_line(dbms_spm.alter_sql_plan_baseline(plan_name=>i.plan_name,attribute_name=>'enabled',attribute_value=>'no')); end if;
5 if i.n in (5,6,7,8) then dbms_output.put_line(dbms_spm.alter_sql_plan_baseline(plan_name=>i.plan_name,attribute_name=>'fixed',attribute_value=>'yes')); end if;
6 for p in ( select plan_table_output from dbms_xplan.display_sql_plan_baseline(plan_name=>i.plan_name,format=>'basic') where plan_table_output like '%| DEMO%' ) loop
7 dbms_output.put_line(dbms_spm.alter_sql_plan_baseline(plan_name=>i.plan_name,attribute_name=>'plan_name',attribute_value=>p.plan_table_output));
8 end loop;
9 end loop;
10 end;
11 /


So here they are, with their new name:

SQL> select plan_name,enabled,accepted,fixed,executions from dba_sql_plan_baselines;
 
PLAN_NAME ENABLED ACCEPTED FIXED EXECUTIONS
---------------------------------- ------- -------- ----- ----------
| 1 | INDEX RANGE SCAN| DEMO4 | NO NO NO 0
| 1 | INDEX RANGE SCAN| DEMO3 | NO YES NO 0
| 1 | INDEX RANGE SCAN| DEMO2 | YES NO NO 0
| 1 | INDEX RANGE SCAN| DEMO6 | YES YES NO 0
| 1 | INDEX RANGE SCAN| DEMO5 | NO NO YES 0
| 1 | INDEX RANGE SCAN| DEMO1 | NO YES YES 30
| 1 | INDEX RANGE SCAN| DEMO8 | YES NO YES 0
| 1 | INDEX RANGE SCAN| DEMO7 | YES YES YES 0


Fixed plans have priority

I flush the shared pool and run my 240 statements again:

SQL> alter system flush shared_pool;
System FLUSH altered
SQL> exec runplans(8)
PL/SQL procedure successfully completed.


Here is the result in V$SQL, only one plan used for all those executions:

SQL> sqlpb
 
SQL_ID PLAN_HASH_VALUE SUM(EXECUTIONS) SQL_PLAN_BASELINE ID OBJECT_NAME
------------- --------------- --------------- ------------------------------------- -- -----------
gqnkn2d17zjvv 1698325646 240 | 1 | INDEX RANGE SCAN| DEMO7 | 1 DEMO7


The only possible plans are those that are fixed and accepted, which are the DEMO6 and DEMO7 ones. However, fixed plans have a priority, so even when the CBO came with the DEMO6 plan it was not used. When there are fixed enabled accepted SQL Plan Baseline, those are the only one considered.

Enabled and Accepted are used

Now setting all fixed attribute to no:

SQL> begin
2 for i in (select rownum n,plan_name from dba_sql_plan_baselines) loop
3 dbms_output.put_line(dbms_spm.alter_sql_plan_baseline(plan_name=>i.plan_name,attribute_name=>'fixed',attribute_value=>'no'));
4 end loop;
5 end;
6 /
PL/SQL procedure successfully completed.
 
SQL> select plan_name,enabled,accepted,fixed,executions from dba_sql_plan_baselines;
PLAN_NAME ENABLED ACCEPTED FIXED EXECUTIONS
---------------------------------- ------- -------- ----- ----------
| 1 | INDEX RANGE SCAN| DEMO4 | NO NO NO 0
| 1 | INDEX RANGE SCAN| DEMO3 | NO YES NO 0
| 1 | INDEX RANGE SCAN| DEMO2 | YES NO NO 0
| 1 | INDEX RANGE SCAN| DEMO6 | YES YES NO 0
| 1 | INDEX RANGE SCAN| DEMO5 | NO NO NO 0
| 1 | INDEX RANGE SCAN| DEMO1 | NO YES NO 30
| 1 | INDEX RANGE SCAN| DEMO8 | YES NO NO 0
| 1 | INDEX RANGE SCAN| DEMO7 | YES YES NO 0


Here is another run:

SQL> alter system flush shared_pool;
System FLUSH altered.
SQL> exec runplans(8)
PL/SQL procedure successfully completed.
 
SQL> sqlpb
 
SQL_ID PLAN_HASH_VALUE SUM(EXECUTIONS) SQL_PLAN_BASELINE ID OBJECT_NAME
------------- --------------- --------------- ------------------------------------- -- -----------
gqnkn2d17zjvv 1698325646 95 | 1 | INDEX RANGE SCAN| DEMO7 | 1 DEMO7
gqnkn2d17zjvv 3449379882 145 | 1 | INDEX RANGE SCAN| DEMO6 | 1 DEMO6


Now that there are no fixed plans taking the priority, all enabled and accepted plans are possible, but only them.

All possible plans in the baseline but none enabled and accepted

Then what happens when all possible plans are in the SQL Plan Baseline but none of them are both enabled and accepted?

SQL> begin
2 for i in (select rownum n,plan_name from dba_sql_plan_baselines where accepted='YES') loop
3 dbms_output.put_line(dbms_spm.alter_sql_plan_baseline(plan_name=>i.plan_name,attribute_name=>'enabled',attribute_value=>'no'));
4 end loop;
5 end;
6 /
 
SQL> select plan_name,enabled,accepted,fixed,executions from dba_sql_plan_baselines;
 
PLAN_NAME ENABLED ACCEPTED FIXED EXECUTIONS
---------------------------------- ------- -------- ----- ----------
| 1 | INDEX RANGE SCAN| DEMO4 | NO NO NO 0
| 1 | INDEX RANGE SCAN| DEMO3 | NO YES NO 0
| 1 | INDEX RANGE SCAN| DEMO2 | YES NO NO 0
| 1 | INDEX RANGE SCAN| DEMO6 | NO YES NO 0
| 1 | INDEX RANGE SCAN| DEMO5 | NO NO NO 0
| 1 | INDEX RANGE SCAN| DEMO1 | NO YES NO 30
| 1 | INDEX RANGE SCAN| DEMO8 | YES NO NO 0
| 1 | INDEX RANGE SCAN| DEMO7 | NO YES NO 0


So all combinations of indexes are there (and my query forces index access with a hint) but none are accepted and enabled.

SQL> alter system flush shared_pool;
System FLUSH altered.
SQL> exec runplans(8)
PL/SQL procedure successfully completed.
 
SQL> sqlpb
SQL_ID PLAN_HASH_VALUE SUM(EXECUTIONS) SQL_PLAN_BASELINE ID OBJECT_NAME
------------- --------------- --------------- -------------------- -- -----------
gqnkn2d17zjvv 3739632713 8 1 DEMO1
gqnkn2d17zjvv 4234411015 16 1 DEMO2
gqnkn2d17zjvv 2199479965 24 1 DEMO3
gqnkn2d17zjvv 1698325646 30 1 DEMO7
gqnkn2d17zjvv 3449379882 30 1 DEMO6
gqnkn2d17zjvv 2144220082 30 1 DEMO5
gqnkn2d17zjvv 918903766 30 1 DEMO4
gqnkn2d17zjvv 39208404 72 1 DEMO8


When there are no enabled and accepted plans, then anything is possible and each execution keeps the one the CBO came with.

None enabled and accepted, but new plan possible

Now, in order to have a new plan possible I’ll still run the same query but after dropping all indexes.

SQL> exec for i in 1..8 loop execute immediate 'drop index DEMO'||i; end loop;
PL/SQL procedure successfully completed.


I’ve run the same as before but without the dbms_stats calls.

Here all executions have run with the only possible plan: a full table scan:

SQL> select sql_id,plan_hash_value,sum(executions),sql_plan_baseline,id,object_name from v$sql join v$sql_plan using(sql_id,plan_hash_value,child_number) where sql_text like 'SELECT % FROM DEMO%' and id=1 group by sql_id,plan_hash_value,sql_plan_baseline,id,object_name order by 3
 
SQL_ID PLAN_HASH_VALUE SUM(EXECUTIONS) SQL_PLAN_BASELINE ID OBJECT_NAME
------------- --------------- --------------- -------------------- -- -----------
gqnkn2d17zjvv 4000794843 29 1 DEMO


this plan has been added, enabled but not accepted, to the SQL Plan Baseline:

SQL> select plan_name,enabled,accepted,fixed,executions from dba_sql_plan_baselines;
 
PLAN_NAME ENABLED ACCEPTED FIXED EXECUTIONS
------------- --------------- --------------- -------------------- -- -----------
| 1 | INDEX RANGE SCAN| DEMO4 | NO NO NO 0
| 1 | INDEX RANGE SCAN| DEMO3 | NO YES NO 0
| 1 | INDEX RANGE SCAN| DEMO2 | YES NO NO 0
| 1 | INDEX RANGE SCAN| DEMO6 | NO YES NO 0
| 1 | INDEX RANGE SCAN| DEMO5 | NO NO NO 0
SQL_PLAN_dcc9d14j7k1vu838f84a8 YES NO NO 0
| 1 | INDEX RANGE SCAN| DEMO1 | NO YES NO 30
| 1 | INDEX RANGE SCAN| DEMO8 | YES NO NO 0
| 1 | INDEX RANGE SCAN| DEMO7 | NO YES NO 0


Not accepted means that it cannot be used. But as there are no other plan possible, it will be used anyway.

In summary:
Fixed plans are like telling to the optimizer: You must use one of these.
Enabled accepted plans are like telling the optimizer: You should use one of these.
Disabled or non-accepted plans are like telling the optimizer: Try to find another plan.
The optimizer will always come with a plan, so if the rules cannot be applied, the optimizer best-estimated plan is used. It may be a non-enabled or non-accepted plan. Or it can be a new plan, which will then be added as non accepted.

Cet article Enabled, Accepted, Fixed SQL Plan Baselines est apparu en premier sur Blog dbi services.

In multitenant, the recovery and availability are at CDB level. But customers asked for a switchover at PDB level so Oracle has done that in 18c, based on refreshable PDBs.

For this test I have two multitenant database on an Oracle Cloud service in 18c: CDB1 and CDB2. The only special thing I did was disable the mandatory TDE encryption, because I was not able to have the switchover working. With TDE encryption, I got the “ORA-46697: Keystore password required”. But there is no ‘keystore identified by’ option in the ‘alter pluggable database’. Then If you came upon this post from a search on this error, I’ve no solution yet (SR 3-17001228251 opened on the Oracle Cloud Support ).

Creating the source PDB

In CDB1 I create CDB1DEMO pluggable database:

21:06:06 //localhost/CDB1 SQL>create pluggable database CDB1DEMO admin user admin identified by oracle file_name_convert=('/pdbseed/','/CDB1DEMO/');
Pluggable database CDB1DEMO created.


I could use ‘show pdb’ but I use SQLcl and the current version (17.4) does not recognize Oracle 18c as a container database, so I’ve created my own alias to run a query onV$PDBS and DBA_PDBS:

21:06:12 //localhost/CDB1 SQL>select con_id,name con_name,open_mode,restricted,status,foreign_cdb_dbid,foreign_pdb_id,to_char(creation_time, 'hh24:mi:ss') creation_time,refresh_mode,refresh_interval,last_refresh_scn,cdb_dbid,current_scn from v$pdbs right outer join dba_pdbs using(con_id,creation_time,dbid) cross join (select dbid cdb_dbid,current_scn from v$database) order by pdb_id;
 
CON_ID CON_NAME OPEN_MODE RESTRICTED STATUS FOREIGN_CDB_DBID FOREIGN_PDB_ID CREATION_TIME REFRESH_MODE REFRESH_INTERVAL LAST_REFRESH_SCN CDB_DBID CURRENT_SCN
------ -------- --------- ---------- ------ ---------------- -------------- ------------- ------------ ---------------- ---------------- -------- -----------
2 PDB$SEED READ ONLY NO NORMAL 1214140 2 12:43:46 NONE 944121613 1524290
3 CDB1DEMO MOUNTED NEW 944121613 2 21:06:06 NONE 944121613 1524290


CDB1DEMO is in mount state in V$PDBS and status is NEW because just created. You may wonder why I ‘right join’ here as PDBs known by the database should always be known by the instance. But I said that I opened a SR for switchover with TDE and here I got an inconsistency between V$PDBS and DBA_PDBS.


21:06:12 //localhost/CDB1 SQL>alter pluggable database CDB1DEMO open;
Pluggable database CDB1DEMO altered.
21:06:14 //localhost/CDB1 SQL>select * from pdb_plug_in_violations;
no rows selected
 
21:06:15 //localhost/CDB1 SQL>select con_id,name con_name,open_mode,restricted,status,foreign_cdb_dbid,foreign_pdb_id,to_char(creation_time, 'hh24:mi:ss') creation_time,refresh_mode,refresh_interval,last_refresh_scn,cdb_dbid,current_scn from v$pdbs right join dba_pdbs using(con_id,creation_time,dbid) cross join (select dbid cdb_dbid,current_scn from v$database) order by pdb_id
 
CON_ID CON_NAME OPEN_MODE RESTRICTED STATUS FOREIGN_CDB_DBID FOREIGN_PDB_ID CREATION_TIME REFRESH_MODE REFRESH_INTERVAL LAST_REFRESH_SCN CDB_DBID CURRENT_SCN
------ -------- --------- ---------- ------ ---------------- -------------- ------------- ------------ ---------------- ---------------- -------- -----------
2 PDB$SEED READ ONLY NO NORMAL 1214140 2 12:43:46 NONE 944121613 1524741
3 CDB1DEMO READ WRITE NO NORMAL 944121613 2 21:06:06 NONE 944121613 1524741


The CDB1DEMO PDB is opened READ WRITE. As my goal is to show refreshes, I need to have database with updates. To do it autonomously I create a materialized view refreshing its timestamp every second.


21:06:15 //localhost/CDB1 SQL>create materialized view DEMO refresh complete start with (sysdate) next (sysdate+1/24/60/60) as select current_timestamp "refresh timestamp",current_scn "refresh scn" from v$database;
Materialized view DEMO created.


Here is how I’ll query this autonomous materialized view, comparing the timestamp and SCN at the time of refresh with the current ones:

21:06:16 //localhost/CDB1 SQL>select demo.*,ora_rowscn,current_timestamp,(select current_scn from v$database) current_scn from demo;
 
refresh timestamp refresh scn ORA_ROWSCN CURRENT_TIMESTAMP CURRENT_SCN
------- --------- ------- --- ---------- ----------------- -----------
10-MAR-18 09.06.16.010529000 PM EUROPE/ZURICH 1524747 1524749 10-MAR-18 09.06.16.560146000 PM EUROPE/ZURICH 1524848


Refreshable PDB refreshing every minute

On CDB1 I create the user I’ll use for the remote clone: C##SYSOPER which needs either SYSOPER or CREATE PLUGGABLE DATABASE privilege.

21:06:17 //localhost/CDB1 SQL>connect sys/oracle@//localhost/CDB1 as sysdba
Connected.
21:06:17 //localhost/CDB1 SQL>grant create session, sysoper to C##SYSOPER identified by oracle1 container=all;
Grant succeeded.
 
21:06:18 //localhost/CDB1 SQL>select con_id,name con_name,open_mode,restricted,status,foreign_cdb_dbid,foreign_pdb_id,to_char(creation_time, 'hh24:mi:ss') creation_time,refresh_mode,refresh_interval,last_refresh_scn,cdb_dbid,current_scn from v$pdbs right join dba_pdbs using(con_id,creation_time,dbid) cross join (select dbid cdb_dbid,current_scn from v$database) order by pdb_id
 
CON_ID CON_NAME OPEN_MODE RESTRICTED STATUS FOREIGN_CDB_DBID FOREIGN_PDB_ID CREATION_TIME REFRESH_MODE REFRESH_INTERVAL LAST_REFRESH_SCN CDB_DBID CURRENT_SCN
------ -------- --------- ---------- ------ ---------------- -------------- ------------- ------------ ---------------- ---------------- -------- -----------
2 PDB$SEED READ ONLY NO NORMAL 1214140 2 12:43:46 NONE 944121613 1524907
3 CDB1DEMO READ WRITE NO NORMAL 944121613 2 21:06:06 NONE 944121613 1524907


On CDB2 I create a database link to connect to this CDB1 user.

21:06:18 //localhost/CDB1 SQL>connect sys/oracle@//localhost/CDB2 as sysdba
Connected.
21:06:18 //localhost/CDB2 SQL>create database link CDB1@SYSOPER connect to C##SYSOPER identified by oracle1 using '//localhost/CDB1'
Database link CDB1@SYSOPER created.
 
21:06:18 //localhost/CDB2 SQL>select con_id,name con_name,open_mode,restricted,status,foreign_cdb_dbid,foreign_pdb_id,to_char(creation_time, 'hh24:mi:ss') creation_time,refresh_mode,refresh_interval,last_refresh_scn,cdb_dbid,current_scn from v$pdbs right join dba_pdbs using(con_id,creation_time,dbid) cross join (select dbid cdb_dbid,current_scn from v$database) order by pdb_id
 
CON_ID CON_NAME OPEN_MODE RESTRICTED STATUS FOREIGN_CDB_DBID FOREIGN_PDB_ID CREATION_TIME REFRESH_MODE REFRESH_INTERVAL LAST_REFRESH_SCN CDB_DBID CURRENT_SCN
------ -------- --------- ---------- ------ ---------------- -------------- ------------- ------------ ---------------- ---------------- -------- -----------
2 PDB$SEED READ ONLY NO NORMAL 1214140 2 12:53:21 NONE 717451787 1522015


Here is the remote clone creating CDB2DEMO from CDB1DEMO, as a refreshable PDB, automatically refreshed every 1 minute (when it is in MOUNT):

21:06:18 //localhost/CDB2 SQL>create pluggable database CDB2DEMO from CDB1DEMO@CDB1@SYSOPER file_name_convert=('/CDB1/','/CDB2/','/CDB1DEMO/','/CDB2DEMO/') refresh mode every 1 minutes;
Pluggable database CDB2DEMO created.


This is not new, we got it in 12cR2 and If you go to Collaborate 18 in Las Vegas next month I’ll demo it: https://app.attendcollaborate.com/event/member/448410 , with all transportable tablespace and pluggable databases data movement.

From the alert.log you can see the clone and one first refresh:

21:06:28 //localhost/CDB2 SQL>host ssh oracle@localhost 'ORACLE_HOME=/u01/app/oracle/product/18.0.0/dbhome_1 /u01/app/oracle/product/18.0.0/dbhome_1/bin/adrci exec="set home diag/rdbms/cdb2/CDB2; show alert -tail 30" '| grep --color -E "(^Completed:|^alter pluggable.*|^create pluggable.*|Media Recovery|onlined Undo|KILL|^Pluggable.*|^)"
2018-03-10 21:06:18.317000 +01:00
create pluggable database CDB2DEMO from CDB1DEMO@CDB1@SYSOPER file_name_convert=('/CDB1/','/CDB2/','/CDB1DEMO/','/CDB2DEMO/') refresh mode every 1 minutes keystore identified by *
Opatch validation is skipped for PDB CDB2DEMO (con_id=4)
2018-03-10 21:06:25.942000 +01:00
Endian type of dictionary set to little
****************************************************************
Pluggable Database CDB2DEMO with pdb id - 4 is created as UNUSABLE.
If any errors are encountered before the pdb is marked as NEW,
then the pdb must be dropped
local undo-1, localundoscn-0x00000000000000fb
****************************************************************
2018-03-10 21:06:27.413000 +01:00
Applying media recovery for pdb-4099 from SCN 1524926 to SCN 1525064
Remote log information: count-1
thr-1, seq-2, logfile-/u01/fast_recovery_area/CDB1/foreign_archivelog/CDB1DEMO/2018_03_10/o1_mf_1_2_212315018_.arc, los-1497297, nxs-18446744073709551615
Media Recovery Start
Serial Media Recovery started
max_pdb is 4
Media Recovery Log /u01/fast_recovery_area/CDB1/foreign_archivelog/CDB1DEMO/2018_03_10/o1_mf_1_2_212315018_.arc
Incomplete Recovery applied until change 1525064 time 03/10/2018 21:06:26
Media Recovery Complete (CDB2)
Completed: create pluggable database CDB2DEMO from CDB1DEMO@CDB1@SYSOPER file_name_convert=('/CDB1/','/CDB2/','/CDB1DEMO/','/CDB2DEMO/') refresh mode every 1 minutes keystore identified by *
alter pluggable database refresh


You can’t open this one in READ WRITE as it is refreshed with the redo from the source, as you see in the alert.log, but you can open it in READ ONLY to query it or to clone it further:

21:06:28 //localhost/CDB2 SQL>alter pluggable database CDB2DEMO open read only;
Pluggable database CDB2DEMO altered.
 
21:06:32 //localhost/CDB2 SQL>alter session set container=CDB2DEMO;
Session altered.
 
21:06:32 //localhost/CDB2 SQL>select demo.*,ora_rowscn,current_timestamp,(select current_scn from v$database) current_scn from demo;
 
refresh timestamp refresh scn ORA_ROWSCN CURRENT_TIMESTAMP CURRENT_SCN
------- --------- ------- --- ---------- ----------------- -----------
10-MAR-18 09.06.27.140229000 PM +01:00 1525071 1525072 10-MAR-18 09.06.32.565600000 PM EUROPE/ZURICH 1525100


if you look at the timestamps, you can see that it is in sync from the source as of the time of the end of creation. The alert.log shows that a refresh happened just after the creation completion.

21:06:39 //localhost/CDB2 SQL>host ssh oracle@localhost 'ORACLE_HOME=/u01/app/oracle/product/18.0.0/dbhome_1 /u01/app/oracle/product/18.0.0/dbhome_1/bin/adrci exec="set home diag/rdbms/cdb2/CDB2; show alert -tail 30" '| grep --color -E "(^Completed:|^alter pluggable.*|^create pluggable.*|Media Recovery|onlined Undo|KILL|^Pluggable.*|^)"
2018-03-10 21:06:28.674000 +01:00
alter pluggable database CDB2DEMO open read only
Applying media recovery for pdb-4099 from SCN 1525064 to SCN 1525083
Remote log information: count-1
thr-1, seq-2, logfile-/u01/fast_recovery_area/CDB1/foreign_archivelog/CDB1DEMO/2018_03_10/o1_mf_1_2_212315018_.arc, los-1497297, nxs-18446744073709551615
Media Recovery Start
Serial Media Recovery started
2018-03-10 21:06:29.721000 +01:00
max_pdb is 4
Media Recovery Log /u01/fast_recovery_area/CDB1/foreign_archivelog/CDB1DEMO/2018_03_10/o1_mf_1_2_212315018_.arc
Incomplete Recovery applied until change 1525083 time 03/10/2018 21:06:28
Media Recovery Complete (CDB2)
Completed: alter pluggable database refresh
Autotune of undo retention is turned on.
2018-03-10 21:06:30.880000 +01:00
Undo initialization finished serial:0 start:4386360 end:4386360 diff:0 ms (0.0 seconds)
Database Characterset for CDB2DEMO is AL32UTF8
2018-03-10 21:06:32.305000 +01:00
Opening pdb with no Resource Manager plan active
Pluggable database CDB2DEMO opened read only
Completed: alter pluggable database CDB2DEMO open read only


The refresh can happen only when the PDB is in MOUNT. If it was opened (READ ONLY) for a long time, it will have to retreive some archive logs. This is why you can see FOREIGN ARCHIVED LOG in your recovery area in 12cR2.

So, basically here you have a kind of standby database at PDB level refreshed asynchonously, that you can open when you want:

21:06:32 //localhost/CDB2 SQL>alter session set container=CDB$ROOT;
Session altered.
21:06:32 //localhost/CDB2 SQL>alter pluggable database CDB2DEMO close;
Pluggable database CDB2DEMO altered.
21:06:39 //localhost/CDB2 SQL>alter pluggable database CDB2DEMO open read only;
Pluggable database CDB2DEMO altered.
21:06:40 //localhost/CDB2 SQL>alter session set container=CDB2DEMO;
Session altered.
21:06:40 //localhost/CDB2 SQL>select demo.*,ora_rowscn,current_timestamp,(select current_scn from v$database) current_scn from demo;
 
refresh timestamp refresh scn ORA_ROWSCN CURRENT_TIMESTAMP CURRENT_SCN
------- --------- ------- --- ---------- ----------------- -----------
10-MAR-18 09.06.27.140229000 PM +01:00 1525071 1525072 10-MAR-18 09.06.40.159432000 PM EUROPE/ZURICH 1525117
 
21:06:40 //localhost/CDB2 SQL>alter session set container=CDB$ROOT;
Session altered.


You will probably do a last refresh before opening it, but I left it on purpose to show that the switchover will ensure that they are in sync.

In 12cR2 if you want to ‘switchover’, you need to stop modifications to CDB1DEMO, alter CDB2DEMO in REFRESH NONE, then open it READ WRITE to be the new ‘primary’ after a last refresh, and drop CDB1DEMO to create it as a refreshable clone from CDB2DEMO. And this is where comes the new 18c PDB Switchover: one command to do all this.

Note that I leave the CDB2DEMO in opened (READ ONLY) or the switchover will fail with ORA-17628: Oracle error 65036 returned by remote Oracle server, ORA-65036: pluggable database not open in required mode.


21:06:40 //localhost/CDB2 SQL>select con_id,name con_name,open_mode,restricted,status,foreign_cdb_dbid,foreign_pdb_id,to_char(creation_time, 'hh24:mi:ss') creation_time,refresh_mode,refresh_interval,last_refresh_scn,cdb_dbid,current_scn from v$pdbs right join dba_pdbs using(con_id,creation_time,dbid) cross join (select dbid cdb_dbid,current_scn from v$database) order by pdb_id
 
CON_ID CON_NAME OPEN_MODE RESTRICTED STATUS FOREIGN_CDB_DBID FOREIGN_PDB_ID CREATION_TIME REFRESH_MODE REFRESH_INTERVAL LAST_REFRESH_SCN CDB_DBID CURRENT_SCN
------ -------- --------- ---------- ------ ---------------- -------------- ------------- ------------ ---------------- ---------------- -------- -----------
2 PDB$SEED READ ONLY NO NORMAL 1214140 2 12:53:21 NONE 717451787 1525131
4 CDB2DEMO READ ONLY NO REFRESHING 944121613 3 21:06:18 AUTO 1 1525083 717451787 1525131


PDB Switchover

As the goal is to change roles, I need a SYSOPER user and a database link on the other sides:

A user to connect to CDB2 which is the source we will refresh from after the switchover:

21:06:40 //localhost/CDB2 SQL>connect sys/oracle@//localhost/CDB2 as sysdba
Connected.
21:06:40 //localhost/CDB2 SQL>grant create session, sysoper to C##SYSOPER identified by oracle2 container=all;
Grant succeeded.


A database link from the CDB1 which will become the refreshing side:

21:06:41 //localhost/CDB2 SQL>connect sys/oracle@//localhost/CDB1 as sysdba
Connected.
21:06:41 //localhost/CDB1 SQL> create database link CDB2@SYSOPER connect to C##SYSOPER identified by oracle2 using '//localhost/CDB2'
Database link CDB2@SYSOPER created.


For the moment this side is in READ WRITE as it is the current ‘primary’

21:06:41 //localhost/CDB1 SQL>select con_id,name con_name,open_mode,restricted,status,foreign_cdb_dbid,foreign_pdb_id,to_char(creation_time, 'hh24:mi:ss') creation_time,refresh_mode,refresh_interval,last_refresh_scn,cdb_dbid,current_scn from v$pdbs right join dba_pdbs using(con_id,creation_time,dbid) cross join (select dbid cdb_dbid,current_scn from v$database) order by pdb_id
 
CON_ID CON_NAME OPEN_MODE RESTRICTED STATUS FOREIGN_CDB_DBID FOREIGN_PDB_ID CREATION_TIME REFRESH_MODE REFRESH_INTERVAL LAST_REFRESH_SCN CDB_DBID CURRENT_SCN
------ -------- --------- ---------- ------ ---------------- -------------- ------------- ------------ ---------------- ---------------- -------- -----------
2 PDB$SEED READ ONLY NO NORMAL 1214140 2 12:43:46 NONE 944121613 1525456
3 CDB1DEMO READ WRITE NO NORMAL 944121613 2 21:06:06 NONE 944121613 1525456


Here is the one-command refresh in 18c. We alter CDB1DEMO to be refreshable from CDB2DEMO, and we add ‘switchover’ to stop refreshing the remote PDB as it will be now the ‘primary’.

21:06:41 //localhost/CDB1 SQL>alter pluggable database CDB1DEMO refresh mode every 1 minutes from CDB2DEMO@CDB2@SYSOPER switchover;
Pluggable database CDB1DEMO altered.


The alert.log here from CDB1 shows ‘Deleted file’ as in a DROP PLUGGABLE DATABASE, then ‘created as UNUSABLE’ as in CREATE PLUGGABLE DATABASE, then ‘Applying media recovery’ as in refreshable clone:

21:06:59 //localhost/CDB1 SQL>host ssh oracle@localhost 'ORACLE_HOME=/u01/app/oracle/product/18.0.0/dbhome_1 /u01/app/oracle/product/18.0.0/dbhome_1/bin/adrci exec="set home diag/rdbms/cdb1/CDB1; show alert -tail 30" '| grep --color -E "(^Completed:|^alter pluggable.*|^create pluggable.*|Media Recovery|onlined Undo|KILL|^Pluggable.*|^)"
2018-03-10 21:06:41.354000 +01:00
alter pluggable database CDB1DEMO refresh mode every 1 minutes from CDB2DEMO@CDB2@SYSOPER switchover
JIT: pid 11896 requesting stop
Buffer Cache flush started: 3
Buffer Cache flush finished: 3
While transitioning the pdb 3 to clean state, clearing all its abort bits in the control file.
Pluggable database CDB1DEMO closed
2018-03-10 21:06:45.734000 +01:00
Deleted file /u01/oradata/CDB1/CDB1DEMO/temp012018-03-10_12-43-46-436-PM.dbf
Deleted file /u01/oradata/CDB1/CDB1DEMO/undotbs01.dbf
Deleted file /u01/oradata/CDB1/CDB1DEMO/sysaux01.dbf
Deleted file /u01/oradata/CDB1/CDB1DEMO/system01.dbf
2018-03-10 21:06:48.199000 +01:00
Opatch validation is skipped for PDB CDB1DEMO (con_id=3)
2018-03-10 21:06:55.321000 +01:00
Endian type of dictionary set to little
****************************************************************
Pluggable Database CDB1DEMO with pdb id - 3 is created as UNUSABLE.
If any errors are encountered before the pdb is marked as NEW,
then the pdb must be dropped
local undo-1, localundoscn-0x00000000000000fb
****************************************************************
2018-03-10 21:06:59.142000 +01:00
Applying media recovery for pdb-4099 from SCN 1526441 to SCN 1526451
Remote log information: count-1
thr-1, seq-2, logfile-/u01/fast_recovery_area/CDB2/foreign_archivelog/CDB2DEMO/2018_03_10/o1_mf_1_2_2195948769_.arc, los-1497207, nxs-18446744073709551615
Media Recovery Start
Serial Media Recovery started
max_pdb is 4
Media Recovery Log /u01/fast_recovery_area/CDB2/foreign_archivelog/CDB2DEMO/2018_03_10/o1_mf_1_2_2195948769_.arc
Incomplete Recovery applied until change 1526451 time 03/10/2018 21:06:55
Media Recovery Complete (CDB1)
Completed: alter pluggable database CDB1DEMO refresh mode every 1 minutes from CDB2DEMO@CDB2@SYSOPER switchover
alter pluggable database refresh


The CDB1DEMO which was the ‘primary’ in READ WRITE is now the ‘standby’ in MOUNT, automatically refreshing every minute as mentioned in the switchover command:

21:07:00 //localhost/CDB1 SQL>select con_id,name con_name,open_mode,restricted,status,foreign_cdb_dbid,foreign_pdb_id,to_char(creation_time, 'hh24:mi:ss') creation_time,refresh_mode,refresh_interval,last_refresh_scn,cdb_dbid,current_scn from v$pdbs right join dba_pdbs using(con_id,creation_time,dbid) cross join (select dbid cdb_dbid,current_scn from v$database) order by pdb_id
 
CON_ID CON_NAME OPEN_MODE RESTRICTED STATUS FOREIGN_CDB_DBID FOREIGN_PDB_ID CREATION_TIME REFRESH_MODE REFRESH_INTERVAL LAST_REFRESH_SCN CDB_DBID CURRENT_SCN
------ -------- --------- ---------- ------ ---------------- -------------- ------------- ------------ ---------------- ---------------- -------- -----------
2 PDB$SEED READ ONLY NO NORMAL 1214140 2 12:43:46 NONE 944121613 1526489
3 CDB1DEMO MOUNTED REFRESHING 717451787 4 21:06:06 AUTO 1 1526451 944121613 1526489


As any refreshable clone, I can open it READ ONLY and query it:

21:07:00 //localhost/CDB1 SQL>alter pluggable database CDB1DEMO open read only;
Pluggable database CDB1DEMO altered.
 
21:07:02 //localhost/CDB1 SQL>alter session set container=CDB1DEMO;
Session altered.
 
21:07:02 //localhost/CDB1 SQL>select demo.*,ora_rowscn,current_timestamp,(select current_scn from v$database) current_scn from demo;
 
refresh timestamp refresh scn ORA_ROWSCN CURRENT_TIMESTAMP CURRENT_SCN
------- --------- ------- --- ---------- ----------------- -----------
10-MAR-18 09.06.41.175918000 PM +01:00 1525436 1525594 10-MAR-18 09.07.02.875782000 PM EUROPE/ZURICH 1526520


Look at the timestamp: the data is freshed as of the switchover. No data is lost: the transactions that were committed on the source at the time of switchover are applied on the clone.

Another switchover

I’m now doing a switchover on the opposite way. Same as before: the destination is READ ONLY and the source is READ WRITE:

21:07:20 //localhost/CDB1 SQL>connect sys/oracle@//localhost/CDB2 as sysdba
Connected.
21:07:20 //localhost/CDB2 SQL>select con_id,name con_name,open_mode,restricted,status,foreign_cdb_dbid,foreign_pdb_id,to_char(creation_time, 'hh24:mi:ss') creation_time,refresh_mode,refresh_interval,last_refresh_scn,cdb_dbid,current_scn from v$pdbs right join dba_pdbs using(con_id,creation_time,dbid) cross join (select dbid cdb_dbid,current_scn from v$database) order by pdb_id
 
CON_ID CON_NAME OPEN_MODE RESTRICTED STATUS FOREIGN_CDB_DBID FOREIGN_PDB_ID CREATION_TIME REFRESH_MODE REFRESH_INTERVAL LAST_REFRESH_SCN CDB_DBID CURRENT_SCN
------ -------- --------- ---------- ------ ---------------- -------------- ------------- ------------ ---------------- ---------------- -------- -----------
2 PDB$SEED READ ONLY NO NORMAL 1214140 2 12:53:21 NONE 717451787 1526851
4 CDB2DEMO READ WRITE NO NORMAL 944121613 3 21:06:18 NONE 717451787 1526851


Here is the switchover

21:07:20 //localhost/CDB2 SQL>alter pluggable database CDB2DEMO refresh mode every 1 minutes from CDB1DEMO@CDB1@SYSOPER switchover;
Pluggable database CDB2DEMO altered.


The alert.log shows an error at the end on the last refresh after the switchover:

21:07:33 //localhost/CDB2 SQL>host ssh oracle@localhost 'ORACLE_HOME=/u01/app/oracle/product/18.0.0/dbhome_1 /u01/app/oracle/product/18.0.0/dbhome_1/bin/adrci exec="set home diag/rdbms/cdb2/CDB2; show alert -tail 30" '| grep --color -E "(^Completed:|^alter pluggable.*|^create pluggable.*|Media Recovery|onlined Undo|KILL|^Pluggable.*|^)"
2018-03-10 21:07:32.707000 +01:00
Incomplete Recovery applied until change 1527253 time 03/10/2018 21:07:31
Media Recovery Complete (CDB2)
Completed: alter pluggable database CDB2DEMO refresh mode every 1 minutes from CDB1DEMO@CDB1@SYSOPER switchover
alter pluggable database refresh
ORA-65376 signalled during: alter pluggable database refresh...
Errors in file /u01/app/oracle/diag/rdbms/cdb2/CDB2/trace/CDB2_j000_12081.trc:
ORA-12012: error on auto execute of job "SYS"."CDB2DEMO_510111146_REFRESH"
ORA-65376: unable to refresh the PDB
ORA-06512: at "SYS.DBMS_SQL", line 2995
ORA-06512: at line 1


The content of the tracefile doesn’t tell a lot more:

ORA-12012: error on auto execute of job "SYS"."CDB2DEMO_510111146_REFRESH"
at 0x7ffd3c59af38 placed jslv.c@1659
ORA-65376: unable to refresh the PDB
ORA-06512: at "SYS.DBMS_SQL", line 2995
ORA-06512: at line 1


However, the switchover was ok, so nothing was lost and I’ll be able to run new refreshes later.

The CDB2DEMO is now the ‘standby’ again:

21:07:33 //localhost/CDB2 SQL>select con_id,name con_name,open_mode,restricted,status,foreign_cdb_dbid,foreign_pdb_id,to_char(creation_time, 'hh24:mi:ss') creation_time,refresh_mode,refresh_interval,last_refresh_scn,cdb_dbid,current_scn from v$pdbs right join dba_pdbs using(con_id,creation_time,dbid) cross join (select dbid cdb_dbid,current_scn from v$database) order by pdb_id
 
CON_ID CON_NAME OPEN_MODE RESTRICTED STATUS FOREIGN_CDB_DBID FOREIGN_PDB_ID CREATION_TIME REFRESH_MODE REFRESH_INTERVAL LAST_REFRESH_SCN CDB_DBID CURRENT_SCN
------ -------- --------- ---------- ------ ---------------- -------------- ------------- ------------ ---------------- ---------------- -------- -----------
2 PDB$SEED READ ONLY NO NORMAL 1214140 2 12:53:21 NONE 717451787 1527275
4 CDB2DEMO MOUNTED REFRESHING 944121613 3 21:06:18 AUTO 1 1527253 717451787 1527275


The CDB1 DEMO is now the ‘primary’ in READ WRITE':

21:07:34 //localhost/CDB2 SQL>connect sys/oracle@//localhost/CDB1 as sysdba
Connected.
21:07:34 //localhost/CDB1 SQL>pdbs
21:07:34 //localhost/CDB1 SQL>select con_id,name con_name,open_mode,restricted,status,foreign_cdb_dbid,foreign_pdb_id,to_char(creation_time, 'hh24:mi:ss') creation_time,refresh_mode,refresh_interval,last_refresh_scn,cdb_dbid,current_scn from v$pdbs right join dba_pdbs using(con_id,creation_time,dbid) cross join (select dbid cdb_dbid,current_scn from v$database) order by pdb_id
 
CON_ID CON_NAME OPEN_MODE RESTRICTED STATUS FOREIGN_CDB_DBID FOREIGN_PDB_ID CREATION_TIME REFRESH_MODE REFRESH_INTERVAL LAST_REFRESH_SCN CDB_DBID CURRENT_SCN
------ -------- --------- ---------- ------ ---------------- -------------- ------------- ------------ ---------------- ---------------- -------- -----------
2 PDB$SEED READ ONLY NO NORMAL 1214140 2 12:43:46 NONE 944121613 1527257
3 CDB1DEMO READ WRITE NO NORMAL 717451787 4 21:06:06 NONE 944121613 1527257


The standby is refreshing, containing the data as-of the time of switchover, until ne next refresh:

21:07:34 //localhost/CDB1 SQL>connect sys/oracle@//localhost/CDB2 as sysdba
Connected.
21:07:35 //localhost/CDB2 SQL>select con_id,name con_name,open_mode,restricted,status,foreign_cdb_dbid,foreign_pdb_id,to_char(creation_time, 'hh24:mi:ss') creation_time,refresh_mode,refresh_interval,last_refresh_scn,cdb_dbid,current_scn from v$pdbs right join dba_pdbs using(con_id,creation_time,dbid) cross join (select dbid cdb_dbid,current_scn from v$database) order by pdb_id
 
CON_ID CON_NAME OPEN_MODE RESTRICTED STATUS FOREIGN_CDB_DBID FOREIGN_PDB_ID CREATION_TIME REFRESH_MODE REFRESH_INTERVAL LAST_REFRESH_SCN CDB_DBID CURRENT_SCN
------ -------- --------- ---------- ------ ---------------- -------------- ------------- ------------ ---------------- ---------------- -------- -----------
2 PDB$SEED READ ONLY NO NORMAL 1214140 2 12:53:21 NONE 717451787 1527276
4 CDB2DEMO MOUNTED REFRESHING 944121613 3 21:06:18 AUTO 1 1527253 717451787 1527276
 
21:07:35 //localhost/CDB2 SQL>alter session set container=CDB2DEMO;
Session altered.
 
21:07:36 //localhost/CDB2 SQL>select demo.*,ora_rowscn,current_timestamp,(select current_scn from v$database) current_scn from demo;
 
refresh timestamp refresh scn ORA_ROWSCN CURRENT_TIMESTAMP CURRENT_SCN
------- --------- ------- --- ---------- ----------------- -----------
10-MAR-18 09.07.20.108426000 PM +01:00 1526838 1526839 10-MAR-18 09.07.36.114424000 PM EUROPE/ZURICH 1527282


I’ve checked the state the next day. The ‘primary’ PDB had its materlialized view still refreshing every second:

12:44:48 //localhost/CDB2 SQL>connect sys/oracle@//localhost/CDB1 as sysdba
Connected.
12:44:48 //localhost/CDB1 SQL>select con_id,name con_name,open_mode,restricted,status,foreign_cdb_dbid,foreign_pdb_id,to_char(creation_time, 'hh24:mi:ss') creation_time,refresh_mode,refresh_interval,last_refresh_scn,cdb_dbid,current_scn from v$pdbs right join dba_pdbs using(con_id,creation_time,dbid) cross join (select dbid cdb_dbid,current_scn from v$database) order by pdb_id
 
CON_ID CON_NAME OPEN_MODE RESTRICTED STATUS FOREIGN_CDB_DBID FOREIGN_PDB_ID CREATION_TIME REFRESH_MODE REFRESH_INTERVAL LAST_REFRESH_SCN CDB_DBID CURRENT_SCN
------ -------- --------- ---------- ------ ---------------- -------------- ------------- ------------ ---------------- ---------------- -------- -----------
2 PDB$SEED READ ONLY NO NORMAL 1214140 2 12:43:46 NONE 944121613 1767409
4 CDB1DEMO READ WRITE NO NORMAL 717451787 3 12:43:31 NONE 944121613 1767409
 
12:44:48 //localhost/CDB1 SQL>alter session set container=CDB1DEMO;
Session altered.
 
12:44:49 //localhost/CDB1 SQL>select demo.*,ora_rowscn,current_timestamp,(select current_scn from v$database) current_scn from demo;
 
refresh timestamp refresh scn ORA_ROWSCN CURRENT_TIMESTAMP CURRENT_SCN
------- --------- ------- --- ---------- ----------------- -----------
11-MAR-18 12.44.33.078430000 PM +01:00 1766975 1766976 11-MAR-18 12.44.49.085200000 PM EUROPE/ZURICH 1767410


And the ‘standby’ PDB is updating every minute:

12:44:49 //localhost/CDB1 SQL>connect sys/oracle@//localhost/CDB2 as sysdba
Connected.
 
12:44:49 //localhost/CDB2 SQL>select con_id,name con_name,open_mode,restricted,status,foreign_cdb_dbid,foreign_pdb_id,to_char(creation_time, 'hh24:mi:ss') creation_time,refresh_mode,refresh_interval,last_refresh_scn,cdb_dbid,current_scn from v$pdbs right join dba_pdbs using(con_id,creation_time,dbid) cross join (select dbid cdb_dbid,current_scn from v$database) order by pdb_id
 
CON_ID CON_NAME OPEN_MODE RESTRICTED STATUS FOREIGN_CDB_DBID FOREIGN_PDB_ID CREATION_TIME REFRESH_MODE REFRESH_INTERVAL LAST_REFRESH_SCN CDB_DBID CURRENT_SCN
------ -------- --------- ---------- ------ ---------------- -------------- ------------- ------------ ---------------- ---------------- -------- -----------
2 PDB$SEED READ ONLY NO NORMAL 1214140 2 12:53:21 NONE 717451787 1767422
3 CDB2DEMO MOUNTED REFRESHING 944121613 4 12:43:42 AUTO 1 1767399 717451787 1767422
 
12:44:49 //localhost/CDB2 SQL>alter session set container=CDB2DEMO;
Session altered.
 
12:44:49 //localhost/CDB2 SQL>alter pluggable database open read only;
Pluggable database OPEN altered.
 
12:44:50 //localhost/CDB2 SQL>select demo.*,ora_rowscn,current_timestamp,(select current_scn from v$database) current_scn from demo;
 
refresh timestamp refresh scn ORA_ROWSCN CURRENT_TIMESTAMP CURRENT_SCN
------- --------- ------- --- ---------- ----------------- -----------
11-MAR-18 12.44.33.078430000 PM +01:00 1766975 1766976 11-MAR-18 12.44.50.205050000 PM EUROPE/ZURICH 1767432


But no failover

When we are talking about ‘standby’ at PDB level, we want to be able to do a failover. Of course, we accept to loose some transactions as the refresh is every minutes, but 1 minute RPO is still an interesting solution for a feature that is available in all editions.

However, as in 12.2.0.1, this doesn’t work because you cannot alter ‘standby’ PDB to REFRESH MODE NONE when the ‘primary’ is not available.

Here is my ‘standby’ PDB CDB2DEMO:

12:44:50 //localhost/CDB2 SQL>connect sys/oracle@//localhost/CDB2 as sysdba
Connected.
 
12:44:50 //localhost/CDB2 SQL>select con_id,name con_name,open_mode,restricted,status,foreign_cdb_dbid,foreign_pdb_id,to_char(creation_time, 'hh24:mi:ss') creation_time,refresh_mode,refresh_interval,last_refresh_scn,cdb_dbid,current_scn from v$pdbs right join dba_pdbs using(con_id,creation_time,dbid) cross join (select dbid cdb_dbid,current_scn from v$database) order by pdb_id
 
CON_ID CON_NAME OPEN_MODE RESTRICTED STATUS FOREIGN_CDB_DBID FOREIGN_PDB_ID CREATION_TIME REFRESH_MODE REFRESH_INTERVAL LAST_REFRESH_SCN CDB_DBID CURRENT_SCN
------ -------- --------- ---------- ------ ---------------- -------------- ------------- ------------ ---------------- ---------------- -------- -----------
2 PDB$SEED READ ONLY NO NORMAL 1214140 2 12:53:21 NONE 717451787 1767433
3 CDB2DEMO READ ONLY NO REFRESHING 944121613 4 12:43:42 AUTO 1 1767405 717451787 1767433


I stop the listener which listens for the ‘primary’ CDB1DEMO:

12:44:50 //localhost/CDB2 SQL>host lsnrctl stop
 
LSNRCTL for Linux: Version 18.0.0.0.0 - Production on 11-MAR-2018 12:44:50
 
Copyright (c) 1991, 2017, Oracle. All rights reserved.
 
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
The command completed successfully


You can see in the alert.log that the refresh fails (it is running from a scheduler job):

Errors in file /u01/app/oracle/diag/rdbms/cdb2/CDB2/trace/CDB2_j000_25443.trc:
ORA-12012: error on auto execute of job "SYS"."CDB2DEMO_1159316120_REFRESH"
ORA-17627: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
ORA-17629: Cannot connect to the remote database server
ORA-06512: at "SYS.DBMS_SQL", line 2995
ORA-06512: at line 1


Then, an attempt to stop the refreshing mode of the ‘standby’ fails with ‘Cannot connect to the remote database server':

12:44:50 //localhost/CDB2 SQL>alter pluggable database CDB2DEMO close;
Pluggable database CDB2DEMO altered.
 
12:44:50 //localhost/CDB2 SQL>alter pluggable database CDB2DEMO refresh mode none;
 
Error starting at line : 180 File @ ~/PDB-switchover.sql
In command -
alter pluggable database CDB2DEMO refresh mode none
Error report -
ORA-17627: ORA-12541: TNS:no listener
ORA-17629: Cannot connect to the remote database server
17627. 00000 - "%s"
*Cause: An error returned by OCI while sending/receiving message from remote instance
*Action: Look at error message and take appropriate action or contact Oracle Support Services for further assistance


So nothing new here about failover. I already explained how to do something like a failover by cloning the standby, which can be a snapshot clone to be faster: https://blog.dbi-services.com/12cr2-pdb-refresh-as-a-poor-man-standby/.

Note that this new feature is leashed to specific platforms only – Oracle Cloud PaaS and Oracle Exadata machine, so most of Oracle customers paying for software update will not be able to use it. However, Don’t worry, you can do the same with a few commands, as in 12cR2.

Cet article 18c PDB switchover est apparu en premier sur Blog dbi services.

I recently wrote a couple of sql-scripts which had to run on all of my customer’s DBs. The sql-scripts had to be started from a Linux-client, which does not have any Oracle client software installed. So I thought of using sqlcl (see http://www.oracle.com/technetwork/developer-tools/sqlcl/downloads/index.html), because there is no need to “install” something then. All I needed was an installed JRE on the Linux-machine. Fortunately that was available. So I downloaded the newest version of sqlcl and unzipped it. Initially I had an issue with the timezone when connecting:


ORA-00604: error occurred at recursive SQL level 1
ORA-01882: timezone region not found


I could workaround this by adding “AddVMOption -Doracle.jdbc.timezoneAsRegion=false” in <sqlcl-home>/bin/sql:


#
# set up the main arguments for java.
#
function setupArgs {
...
AddVMOption -Doracle.jdbc.timezoneAsRegion=false
....


When looping over the databases the connect-times suddenly became very slow. I had to wait for up to 2 minutes to setup a connection to the database. After doing strace on the sqlcl-connects I saw the reason for the slow connects:

The Oracle JDBC driver requires a random number for encrypting the connect string. By default this random number is generated from /dev/random. If the number of entropy-entries (cat /proc/sys/kernel/random/entropy_avail) becomes too low, the access to /dev/random is blocked until enough entropy-entries are available again. The number of entropy-entries is generated through some random noise on the machine (drivers for keyboard, mouse, network, etc. generate the entries). If there is no driver-activity then the entropy-pool may become empty.

The workaround is to
– either artificially generate some driver load (examples on how to do it are in referenced links at the end of the blog)
– or use /dev/urandom instead of /dev/random

The first workaround was not feasible in my case. So I had to use the more unsecure workaround of using /dev/urandom. I.e. I updated <sqlcl-home>/bin/sql again and added “AddVMOption -Djava.security.egd=file:///dev/urandom”:


#
# set up the main arguments for java.
#
function setupArgs {
...
AddVMOption -Djava.security.egd=file:///dev/urandom
....


Afterwards my sqlcl-connects were fast again.

See the following links for more info:

http://www.usn-it.de/index.php/2009/02/20/oracle-11g-jdbc-driver-hangs-blocked-by-devrandom-entropy-pool-empty
https://stackoverflow.com/questions/2327220/oracle-jdbc-intermittent-connection-issue
https://oraganism.wordpress.com/2015/09/20/slow-jdbc-connections-strace-and-random-numbers
http://www.oracle.com/technetwork/database/database-appliance/documentation/oda-eval-comparing-performance-1895230.pdf –> see page 7: “Specify non-blocking random number generators”

Remark: I updated the Oracle Community entry https://community.oracle.com/thread/4073170 with that information as well.

Cet article Connect-times to the DB suddenly become very slow using sqlcl est apparu en premier sur Blog dbi services.

In a previous post about Result Cache, I was using ‘set autotrace on’ in my session to see when the result cache hit saves the logical reads. I got a comment, by e-mail and with full test case, mentioning that the invalidation I had when doing some DML does not occur when autotrace is set to on. I reproduced the problem, on 18c because I use my cloud services to do my tests, but it works the same way in 12c.

I’ll focus here on DML which does not modify any rows from the table the result depends on, because this is the case that depends on the session transaction context. When rows are modified, the result is always invalidated.

For each test here I’ll flush the result cache:

SQL> exec if dbms_result_cache.flush then dbms_output.put_line('Flushed.'); end if;
PL/SQL procedure successfully completed.
 
SQL> select id,type,status,name,cache_id,invalidations from v$result_cache_objects order by id
no rows selected

And I begin each test with the result cache loaded by a query.

DML on the dependent table

The result or the query is loaded into the cache, with a dependency on the SCOTT.DEPT table:

SQL> select id,type,status,name,cache_id,invalidations from v$result_cache_objects order by id
 
ID TYPE STATUS NAME CACHE_ID INVALIDATIONS
-- ---- ------ ---- -------- -------------
0 Dependency Published SCOTT.DEPT SCOTT.DEPT 0
1 Result Published select /*+ result_cache */ count(*) from DEPT 50vtzqa5u0xcy4bnh6z600nj1u 0


Now, I’m executing some DML on this DEPT table:

SQL> connect scott/tiger@//localhost/PDB1
Connected.
SQL> delete from DEPT where DEPTNO>40;
0 rows deleted.

This delete does not touch any row, but declares the intention to modify the table with a Row eXclusive lock (TM mode=3):

SQL> select * from v$locked_object natural join (select object_id,object_name from user_objects)
 
OBJECT_ID XIDUSN XIDSLOT XIDSQN SESSION_ID ORACLE_USERNAME OS_USER_NAME PROCESS LOCKED_MODE CON_ID OBJECT_NAME
--------- ------ ------- ------ ---------- --------------- ------------ ------- ----------- ------ -----------
73376 0 0 0 21 SCOTT oracle 7772 3 5 DEPT

Note that the transaction ID is all zeros. Logically, we are in a transaction, as we have a lock that will be released only at the end of the transaction (commit or rollback). But, as we didn’t modify any block yet, there is no entry in the transaction table:

SQL> select xidusn,xidslot,xidsqn,start_time,used_urec from v$transaction
no rows selected

I commit this transaction:

SQL> commit;
Commit complete.

and check the result cache objects:

SQL> select id,type,status,name,cache_id,invalidations from v$result_cache_objects order by id
 
ID TYPE STATUS NAME CACHE_ID INVALIDATIONS
-- ---- ------ ---- -------- -------------
0 Dependency Published SCOTT.DEPT SCOTT.DEPT 0
1 Result Published select /*+ result_cache */ count(*) from DEPT 50vtzqa5u0xcy4bnh6z600nj1u 0


This look good: nothing was modified so there is no reason to invalidate the result. However, in the previous post I referenced, a similar test invalidated the cache even when no rows were touched. And a reader remarked that this occured only when I previously run a query with ‘autotrace on’. Without autotrace, the behaviour is like what I show here above: no invalidation when no rows is modified.

Same in an already started transaction

The difference is that the ‘autotrace on’ runs an ‘explain plan’, filling the plan table, which means that a transaction was already started. Here I run autotrace on a completely different query:

SQL> set autotrace on explain
Autotrace Enabled
Displays the execution plan only.
SQL> select * from dual;
DUMMY
-----
X
 
Explain Plan
-----------------------------------------------------------
PLAN_TABLE_OUTPUT
Plan hash value: 272002086
 
--------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
--------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 2 | 2 (0)| 00:00:01 |
| 1 | TABLE ACCESS FULL| DUAL | 1 | 2 | 2 (0)| 00:00:01 |
--------------------------------------------------------------------------
 
SQL> set autotrace off;
Autotrace Disabled


Following this, I have no locked objects, but a transaction has been initiated:

SQL> select * from v$locked_object natural join (select object_id,object_name from user_objects)
no rows selected
 
SQL> select xidusn,xidslot,xidsqn,start_time,used_urec from v$transaction
XIDUSN XIDSLOT XIDSQN START_TIME USED_UREC
------ ------- ------ ---------- ---------
9 18 709 03/18/18 20:30:09 2


Here is the big difference: I have a transaction ID here.
Now doing the same as before, a delete touching no rows:

SQL> delete from DEPT where DEPTNO>40;
0 rows deleted.


When I query the lock objects, they are now associated to a non-zero transaction ID:

SQL> select * from v$locked_object natural join (select object_id,object_name from user_objects)
 
OBJECT_ID XIDUSN XIDSLOT XIDSQN SESSION_ID ORACLE_USERNAME OS_USER_NAME PROCESS LOCKED_MODE CON_ID OBJECT_NAME
--------- ------ ------- ------ ---------- --------------- ------------ ------- ----------- ------ -----------
73376 9 18 709 21 SCOTT oracle 7772 3 5 DEPT


Here is the difference, now the result cache has been invalidated:

SQL> commit;
Commit complete.
 
SQL> select id,type,status,name,cache_id,invalidations from v$result_cache_objects order by id
 
ID TYPE STATUS NAME CACHE_ID INVALIDATIONS
-- ---- ------ ---- -------- -------------
0 Dependency Published SCOTT.DEPT SCOTT.DEPT 1
1 Result Invalid select /*+ result_cache */ count(*) from DEPT 50vtzqa5u0xcy4bnh6z600nj1u 0


DML on another table – RS though referential integrity

This gives the idea that the invalidation is not really triggered by actual modifications, but at commit time from the DML locks when associated with a transaction.
And some DML on tables may acquire a Row-S or Row-X lock on the tables linked by referential integrity. This has changed a lot though versions – look at the slide on the right.

I start in the same situation, with the result cache loaded, no locked objects, but a transaction that has started:

SQL> select id,type,status,name,cache_id,invalidations from v$result_cache_objects order by id
 
ID TYPE STATUS NAME CACHE_ID INVALIDATIONS
-- ---- ------ ---- -------- -------------
0 Dependency Published SCOTT.DEPT SCOTT.DEPT 0
1 Result Published select /*+ result_cache */ count(*) from DEPT 50vtzqa5u0xcy4bnh6z600nj1u 0
 
SQL> select * from v$locked_object natural join (select object_id,object_name from user_objects)
no rows selected
 
SQL> select xidusn,xidslot,xidsqn,start_time,used_urec from v$transaction
 
XIDUSN XIDSLOT XIDSQN START_TIME USED_UREC
------ ------- ------ ---------- ---------
8 31 766 03/18/18 20:30:10 2


I delete from the EMP table and if you are familiar with the SCOTT schema, you know that it has a foreign key to DEPT.

SQL> delete from EMP where DEPTNO>=40;
0 rows deleted.


The delete acquires a Row eXclusive lock on EMP even when there are no rows deleted (DML locks are about the intention to modify rows). And from the table above, it acquires a Row Share (mode=2) on the parent table:

SQL> select * from v$locked_object natural join (select object_id,object_name from user_objects)
 
OBJECT_ID XIDUSN XIDSLOT XIDSQN SESSION_ID ORACLE_USERNAME OS_USER_NAME PROCESS LOCKED_MODE CON_ID OBJECT_NAME
--------- ------ ------- ------ ---------- --------------- ------------ ------- ----------- ------ -----------
73376 8 31 766 21 SCOTT oracle 7772 2 5 DEPT
73378 8 31 766 21 SCOTT oracle 7772 3 5 EMP

I commit and check the result cache:

SQL> commit;
Commit complete.
 
SQL> select id,type,status,name,cache_id,invalidations from v$result_cache_objects order by id
 
ID TYPE STATUS NAME CACHE_ID INVALIDATIONS
-- ---- ------ ---- -------- -------------
0 Dependency Published SCOTT.DEPT SCOTT.DEPT 0
1 Result Published select /*+ result_cache */ count(*) from DEPT 50vtzqa5u0xcy4bnh6z600nj1u 0


No invalidation here, which makes sense because Row Share is not an intention to write.

DML on another table – RS though referential integrity

I do the same here, but with an insert on EMP which acquires a Row eXclusive on the parent table.

SQL> insert into EMP(EMPNO,DEPTNO)values(9999,40);
1 row inserted.
 
SQL> select * from v$locked_object natural join (select object_id,object_name from user_objects)
 
OBJECT_ID XIDUSN XIDSLOT XIDSQN SESSION_ID ORACLE_USERNAME OS_USER_NAME PROCESS LOCKED_MODE CON_ID OBJECT_NAME
--------- ------ ------- ------ ---------- --------------- ------------ ------- ----------- ------ -----------
73376 10 32 560 21 SCOTT oracle 7772 3 5 DEPT
73378 10 32 560 21 SCOTT oracle 7772 3 5 EMP
 
SQL> select xidusn,xidslot,xidsqn,start_time,used_urec from v$transaction
 
XIDUSN XIDSLOT XIDSQN START_TIME USED_UREC
------ ------- ------ ---------- ---------
10 32 560 03/18/18 20:30:10 4
 
SQL> commit;
Commit complete.
 
SQL> select id,type,status,name,cache_id,invalidations from v$result_cache_objects order by id
---- ------ ------------------------------------------ ---- ---------------------- ----- -- --
ID TYPE STATUS NAME CACHE_ID INVALIDATIONS
0 Dependency Published SCOTT.DEPT SCOTT.DEPT 1
1 Result Invalid select /*+ result_cache */ count(*) from DEPT 50vtzqa5u0xcy4bnh6z600nj1u 0


Here, DEPT has been invalidated after the insert on EMP. There were no modifications on DEPT, but the result cache is not directly tracking the modifications, but rather the intention of modification. And then, the implementation of the result cache invalidation tracks, at commit, the Row eXclusive locks when they are related to a know transaction. You can have the same result if, from a transaction that has already started, you run a simple:

SQL> lock table DEPT in row exclusive mode;
Lock succeeded.


This means that there are many reasons why the result cache may be invalid even when the objects in the ‘Dependency’ are not modified. Be careful, invalidations and cache misses are expensive and do not scale on high load. And because of locks through referential integrity, this can happen even on static tables. Let’s take an example. in an order entry system, you may think that the ‘products’ table is a good candidate for result cache – updated twice a year but read all the times. But now that you know that inserts on child tables, such as the order table, will invalidate this cache, you may think again about it.

Cet article Result cache invalidation caused by DML locks est apparu en premier sur Blog dbi services.

Here is how I install the latest Docker version on Oracle Linux 7. You find several blog posts about it which all install ‘docker-engine’. But things move fast in this agile world and docker package name has changed. The Community Edition is now ‘docker-ce’ and you want this one to run the latest version.

I’m on OEL 7.4 but should also wotj on RHEL 7:
[root@VM188 yum]# cat /etc/oracle-release
Oracle Linux Server release 7.4

docker-engine

If you enable [ol7_addons] you can install ‘docker-engine:’

# yum-config-manager --enable ol7_addons
# yum info docker-engine
Loaded plugins: ulninfo
Available Packages
Name : docker-engine
Arch : x86_64
Version : 17.06.2.ol
Release : 1.0.1.el7
Size : 21 M
Repo : ol7_addons/x86_64
Summary : The open-source application container engine
URL : https://dockerproject.org
License : ASL 2.0
Description : Docker is an open source project to build, ship and run any application as a
: lightweight container.

But forget it. That’s 17.06 which is 6 months old. You should consider Docker as a puppy. 6 month in puppy’s years is like 3 human years. So many changes happened.

You can remove all those old things:

# yum remove docker docker-common docker-selinux docker-engine


docker-ce

I’ve not found ‘docker-ce’ on OL7 repositories, as only the Enterprise Edition is there. Then I added the CentOS repo (with yum-config-manager that you can get with yum-utils if you don’t have it already):

yum -y install yum-utils
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo


Once done, I’ve just installed Docker Community Edition with:

yum -y install docker-ce


And, at that time I got the version 17.12 easily:

[root@VM188 ~]# yum info docker-ce
Loaded plugins: ulninfo
Installed Packages
Name : docker-ce
Arch : x86_64
Version : 17.12.0.ce
Release : 1.el7.centos
Size : 123 M
Repo : installed
From repo : docker-ce-stable
Summary : The open-source application container engine
URL : https://www.docker.com
License : ASL 2.0
Description : Docker is an open source project to build, ship and run any application as a
: lightweight container.
...


But now there’s a new version available:

Available Packages
Name : docker-ce
Arch : x86_64
Version : 18.03.0.ce
Release : 1.el7.centos
Size : 35 M
Repo : docker-ce-stable/x86_64
Summary : The open-source application container engine
URL : https://www.docker.com
License : ASL 2.0
Description : Docker is an open source project to build, ship and run any application as a
: lightweight container.


The problem is that if you want to install docker-ce in this latest version, you will now get:

Resolving Dependencies
--> Running transaction check
---> Package docker-ce.x86_64 0:18.03.0.ce-1.el7.centos will be installed
--> Processing Dependency: pigz for package: docker-ce-18.03.0.ce-1.el7.centos.x86_64
--> Finished Dependency Resolution
Error: Package: docker-ce-18.03.0.ce-1.el7.centos.x86_64 (docker-ce-stable)
Requires: pigz
You could try using --skip-broken to work around the problem
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
2:microcode_ctl-2.1-22.5.0.3.el7_4.x86_64 has missing requires of kernel

(Ok Google, this is what you need to index…)

pigz

Starting from version 18.02 there’s a new dependency on ‘pigz’ for parallel gzip.

To get this ‘pigz’ package from the OL7 repository you need to enable EPEL in /etc/yum.repos.d/public-yum-ol7.repo

[ol7_developer_EPEL] name=Oracle Linux $releasever Developement Packages ($basearch)
baseurl=http://yum.oracle.com/repo/OracleLinux/OL7/developer_EPEL/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1


Now, I’m able to install the latest docker-ce:

[root@VM188 yum.repos.d]# yum install docker-ce
Loaded plugins: ulninfo
Resolving Dependencies
--> Running transaction check
---> Package docker-ce.x86_64 0:17.12.0.ce-1.el7.centos will be updated
---> Package docker-ce.x86_64 0:18.03.0.ce-1.el7.centos will be an update
--> Processing Dependency: pigz for package: docker-ce-18.03.0.ce-1.el7.centos.x86_64
--> Running transaction check
---> Package pigz.x86_64 0:2.3.4-1.el7 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
================================================================================================================================
Package Arch Version Repository Size
================================================================================================================================
Updating:
docker-ce x86_64 18.03.0.ce-1.el7.centos docker-ce-stable 35 M
Installing for dependencies:
pigz x86_64 2.3.4-1.el7 ol7_developer_EPEL 80 k
 
Transaction Summary
================================================================================================================================
Install ( 1 Dependent package)
Upgrade 1 Package
 
Total download size: 35 M
Is this ok [y/d/N]: y


Oracle Database on Docker

You may wonder why I install Docker on Oracle Linux rather than CentOS. The MOS Doc ID 2216342.1 mentions that Oracle will support customers running Oracle Database (single instance) in Docker containers running on Oracle Linux 7 with UEK4 or Red Hat Enterprise Linux 7.

If you want to validate your Docker install for running Oracle Database, the easiest is to use the image build script provided by Oracle:

git clone https://github.com/oracle/docker-images.git
cd ./docker-images/OracleDatabase/SingleInstance/dockerfiles/
# download and move linuxx64_12201_database.zip is in 12.2.0.1 subdirectory
sh buildDockerImage.sh -v 12.2.0.1 -e

Those are maintained by Gerald Venzl, Oracle product manager for database development, so they are obviously the best way to run Oracle Database on Docker. You can read all related best practices from the same author. Once you have that running, you have validated your environment and you can customize further if you want.

Cet article Docker-CE on Oracle Enterprise Linux 7 est apparu en premier sur Blog dbi services.

I see increasing demand to build a Docker image for the Oracle Database. But the installation process for Oracle does not really fit the Docker way to install by layers: you need to unzip the distribution, install from it to the Oracle Home, remove the things that are not needed, strop the binaries,… Before addressing those specific issues, here are the little tests I’ve done to show how the build layers increase the size of the image.

I’m starting with an empty docker repository on XFS filesystem:

[root@VM121 docker]# df -hT /var/lib/docker
Filesystem Type Size Used Avail Use% Mounted on
/dev/sdc xfs 80G 33M 80G 1% /var/lib/docker


add, copy, rename and append

For the example, I create a 100MB file in the context:

[root@VM121 docker]# mkdir -p /var/tmp/demo
[root@VM121 docker]# dd if=/dev/urandom of=/var/tmp/demo/file0.100M count=100 bs=1M


Here his my docker file:

FROM alpine:latest as staging
WORKDIR /var/tmp
ADD file0.100M .
RUN cp file0.100M file1.100M
RUN rm file0.100M
RUN mv file1.100M file2.100M
RUN dd if=/dev/urandom of=file2.100M seek=100 count=100 bs=1M

The 1st step starts with an alpine image
The 2nd step sets the working directory
The 3rd step adds a 100M file from the context
The 4th step copies the file, so that we have 200M in two files
The 5th step removes the previous file, so that we have 100M in one file
The 6th step renames the file, staying with only one 100M file
The 7th step appends 100M to the file, leaving 200M in one file

Here is the build with default option:

[root@VM121 docker]# docker image build -t franck/demo /var/tmp/demo


The context, my 100M files is send first:

Sending build context to Docker daemon 104.9MB


And here are my 7 steps:

Step 1/7 : FROM alpine:latest as staging
latest: Pulling from library/alpine
ff3a5c916c92: Pull complete
Digest: sha256:7df6db5aa61ae9480f52f0b3a06a140ab98d427f86d8d5de0bedab9b8df6b1c0
Status: Downloaded newer image for alpine:latest
---> 3fd9065eaf02
Step 2/7 : WORKDIR /var/tmp
Removing intermediate container 93d1b5f21bb9
---> 131b3e6f34e7
Step 3/7 : ADD file0.100M .
---> 22ca0b2f6424
Step 4/7 : RUN cp file0.100M file1.100M
---> Running in b4b1b9c7e29b
Removing intermediate container b4b1b9c7e29b
---> 8c7290a5c87e
Step 5/7 : RUN rm file0.100M
---> Running in 606e2c73d456
Removing intermediate container 606e2c73d456
---> 5287e66b019c
Step 6/7 : RUN mv file1.100M file2.100M
---> Running in 10a9b379150e
Removing intermediate container 10a9b379150e
---> f508f426f70e
Step 7/7 : RUN dd if=/dev/urandom of=file2.100M seek=100 count=100 bs=1M
---> Running in 9dcf6d80642c
100+0 records in
100+0 records out
Removing intermediate container 9dcf6d80642c
---> f98304641c54
Successfully built f98304641c54
Successfully tagged franck/demo:latest


So, what’s the size of my docker repository after my image with this 200M file?

[root@VM121 docker]# df -hT /var/lib/docker
Filesystem Type Size Used Avail Use% Mounted on
/dev/sdc xfs 80G 538M 80G 1% /var/lib/docker


I have more than 500MB here.

Actually, besides the alpine image downloaded, which is only 4MB, the image I have build is 538MB:

[root@VM121 docker]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
franck/demo latest f98304641c54 Less than a second ago 528MB
alpine latest 3fd9065eaf02 2 months ago 4.15MB


We can better understand this size by looking at intermediate images:

[root@VM121 docker]# docker image ls -a
REPOSITORY TAG IMAGE ID CREATED SIZE
franck/demo latest f98304641c54 1 second ago 528MB
<none> <none> f508f426f70e 27 seconds ago 319MB
<none> <none> 5287e66b019c 36 seconds ago 214MB
<none> <none> 8c7290a5c87e 37 seconds ago 214MB
<none> <none> 22ca0b2f6424 42 seconds ago 109MB
<none> <none> 131b3e6f34e7 47 seconds ago 4.15MB
alpine latest 3fd9065eaf02 2 months ago 4.15MB


The first one, ’22ca0b2f6424′ is from the step 3 which added the 100MB file
The second one ‘8c7290a5c87e’ is from the 4th step which copied the file, bringing the image to 200MB
The third one ‘5287e66b019c’ is from the 5th step which removed the file. I didn’t increase the size but didn’t remove anything either.
The fourth one ‘f508f426f70e’ is from the 6th step which renamed the file. But this, for docker, is like copying to a new layer and that adds 100MB
Finally, the 7th step appended only 100MB, but this finally resulted to copy the full 200MB file to the new layer

We can see all those operations, and size added at each step, from the image history:

[root@VM121 docker]# docker image history franck/demo
IMAGE CREATED CREATED BY SIZE COMMENT
f98304641c54 1 second ago /bin/sh -c dd if=/dev/urandom of=file2.100M … 210MB
f508f426f70e 27 seconds ago /bin/sh -c mv file1.100M file2.100M 105MB
5287e66b019c 36 seconds ago /bin/sh -c rm file0.100M 0B
8c7290a5c87e 37 seconds ago /bin/sh -c cp file0.100M file1.100M 105MB
22ca0b2f6424 42 seconds ago /bin/sh -c #(nop) ADD file:339435a18aeeb1b69… 105MB
131b3e6f34e7 47 seconds ago /bin/sh -c #(nop) WORKDIR /var/tmp 0B
3fd9065eaf02 2 months ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B
<missing> 2 months ago /bin/sh -c #(nop) ADD file:093f0723fa46f6cdb… 4.15MB


All in one RUN

One workaround is to run everything in the same layer. Personally, I don’t like it because I don’t get the point of using a Dockerfile for just running one script.
So, here is the Dockerfile with only one RUN command:

FROM alpine:latest as staging
WORKDIR /var/tmp
ADD file0.100M .
RUN cp file0.100M file1.100M \
&& rm file0.100M \
&& mv file1.100M file2.100M \
&& dd if=/dev/urandom of=file2.100M seek=100 count=100 bs=1M


The build is similar except that there are fewer steps:

[root@VM121 docker]# docker image build -t franck/demo /var/tmp/demo
Sending build context to Docker daemon 104.9MB
Step 1/4 : FROM alpine:latest as staging
latest: Pulling from library/alpine
ff3a5c916c92: Pull complete
Digest: sha256:7df6db5aa61ae9480f52f0b3a06a140ab98d427f86d8d5de0bedab9b8df6b1c0
Status: Downloaded newer image for alpine:latest
---> 3fd9065eaf02
Step 2/4 : WORKDIR /var/tmp
Removing intermediate container 707644c15547
---> d4528b28c85e
Step 3/4 : ADD file0.100M .
---> e26215766e75
Step 4/4 : RUN cp file0.100M file1.100M && rm file0.100M && mv file1.100M file2.100M && dd if=/dev/urandom of=file2.100M seek=100 count=100 bs=1M
---> Running in 49c2774851f4
100+0 records in
100+0 records out
Removing intermediate container 49c2774851f4
---> df614ac1b6b3
Successfully built df614ac1b6b3
Successfully tagged franck/demo:latest


This leaves us with a smaller space usage::

[root@VM121 docker]# df -hT /var/lib/docker
Filesystem Type Size Used Avail Use% Mounted on
/dev/sdc xfs 80G 340M 80G 1% /var/lib/docker


The image is smaller, but still larger than the final state (a 300MB image for only one 200MB file):

[root@VM121 docker]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
franck/demo latest df614ac1b6b3 Less than a second ago 319MB
alpine latest 3fd9065eaf02 2 months ago 4.15MB


This is because we have grouped the RUN steps, but the ADD has its own layer, adding a file that is removed later:

[root@VM121 docker]# docker image ls -a
REPOSITORY TAG IMAGE ID CREATED SIZE
franck/demo latest df614ac1b6b3 Less than a second ago 319MB
<none> <none> e26215766e75 20 seconds ago 109MB
<none> <none> d4528b28c85e 22 seconds ago 4.15MB
alpine latest 3fd9065eaf02 2 months ago 4.15MB
 
[root@VM121 docker]# docker image history franck/demo
IMAGE CREATED CREATED BY SIZE COMMENT
df614ac1b6b3 Less than a second ago /bin/sh -c cp file0.100M file1.100M … 210MB
e26215766e75 20 seconds ago /bin/sh -c #(nop) ADD file:fe0262a4b800bf66d… 105MB
d4528b28c85e 22 seconds ago /bin/sh -c #(nop) WORKDIR /var/tmp 0B
3fd9065eaf02 2 months ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B
<missing> 2 months ago /bin/sh -c #(nop) ADD file:093f0723fa46f6cdb… 4.15MB


This is the kind of issue we have when building an Oracle Database image. We need to ADD the zip file for the database distribution, and the latest bundle patch. It is removed later but still takes space on the image. Note that one workaround to avoid the ADD layer can be to get the files from an NFS or HTTP server with wget or curl in a RUN layer rather than an ADD one. There’s an example on Stefan Oehrli blog post.

–squash

With the latest versions of docker, there’s an easy way to flatten all those intermediary images at the end.
Here I’ve 18.03 and enabled experimental features:

[root@VM121 docker]# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 8
Server Version: 18.03.0-ce
Storage Driver: overlay2
Backing Filesystem: xfs
...
 
[root@VM121 docker]# cat /etc/docker/daemon.json
{
"experimental": true
}


I start with the same as before but just add –squash to the build command

[root@VM121 docker]# docker image build --squash -t franck/demo /var/tmp/demo


The output is similar but the image is an additional one, reduced down to the size of my final state (with one 200MB file):

[root@VM121 docker]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
franck/demo latest 2ab439a723c4 Less than a second ago 214MB
<none> <none> c3058e598b0a 3 seconds ago 528MB
alpine latest 3fd9065eaf02 2 months ago 4.15MB


The intermediate image list shows that all was done as without ‘–squash’ but with an additional set which reduced the size:

[root@VM121 docker]# docker image ls -a
REPOSITORY TAG IMAGE ID CREATED SIZE
franck/demo latest 2ab439a723c4 Less than a second ago 214MB
<none> <none> c3058e598b0a 3 seconds ago 528MB
<none> <none> 1f14d93a592e 23 seconds ago 319MB
<none> <none> 7563d40b650b 27 seconds ago 214MB
<none> <none> 8ed15a5059bd 28 seconds ago 214MB
<none> <none> 24b11b9026ce 31 seconds ago 109MB
<none> <none> 382bb71a6a4a 33 seconds ago 4.15MB
alpine latest 3fd9065eaf02 2 months ago 4.15MB


This step is visible in the image history as a ‘merge’ step:

[root@VM121 docker]# docker image history franck/demo
IMAGE CREATED CREATED BY SIZE COMMENT
2ab439a723c4 Less than a second ago 210MB merge sha256:c3058e598b0a30c606c1bfae7114957bbc62fca85d6a70c2aff4473726431394 to sha256:3fd9065eaf02feaf94d68376da52541925650b81698c53c6824d92ff63f98353
<missing> 3 seconds ago /bin/sh -c dd if=/dev/urandom of=file2.100M … 0B
<missing> 23 seconds ago /bin/sh -c mv file1.100M file2.100M 0B
<missing> 27 seconds ago /bin/sh -c rm file0.100M 0B
<missing> 28 seconds ago /bin/sh -c cp file0.100M file1.100M 0B
<missing> 31 seconds ago /bin/sh -c #(nop) ADD file:14cef588b48ffbbf1… 0B
<missing> 33 seconds ago /bin/sh -c #(nop) WORKDIR /var/tmp 0B
<missing> 2 months ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B
<missing> 2 months ago /bin/sh -c #(nop) ADD file:093f0723fa46f6cdb… 4.15MB


However, even if I have a smaller final image, my filesystem usage is even larger with this additional 210MB:

[root@VM121 docker]# df -hT /var/lib/docker
Filesystem Type Size Used Avail Use% Mounted on
/dev/sdc xfs 80G 739M 80G 1% /var/lib/docker


Let’s prune it to get rid of those intermediate images:

[root@VM121 docker]# docker image prune -f
Deleted Images:
deleted: sha256:c3058e598b0a30c606c1bfae7114957bbc62fca85d6a70c2aff4473726431394
deleted: sha256:37ed4826d70def1978f9dc0ddf42618d951f65a79ce30767ac3a5037d514f8af
deleted: sha256:1f14d93a592eb49a210ed73bf65e6886fcec332786d54b55d6b0e16fb8a8beda
deleted: sha256:c65cf4c70aed04e9b57e7a2a4fa454d3c63f43c32af251d8c86f6f85f44b1757
deleted: sha256:7563d40b650b2126866e8072b8df92d5d7516d86b25a2f6f99aa101bb47835ba
deleted: sha256:31ee5456431e903cfd384b1cd7ccb7918d203dc73a131d4ff0b9e6517f0d51cd
deleted: sha256:8ed15a5059bd4c0c4ecb78ad77ed75da143b06923d8a9a9a67268c62257b6534
deleted: sha256:6be91d85dec6e1bda6f1c0d565e98dbf928b4ea139bf9cb666455e77a2d8f0d9
deleted: sha256:24b11b9026ce738a78ce3f7b8b5d86ba3fdeb15523a30a7c22fa1e3712ae679a
deleted: sha256:c0984945970276621780a7888adfde9c6e6ca475c42af6b7c54f664ad86f9c9f
deleted: sha256:382bb71a6a4a7ddec86faa76bb86ea0c1a764e5326ad5ef68ce1a6110ae45754
 
Total reclaimed space: 524.3MB


Now having only the squashed image:

[root@VM121 docker]# docker image ls -a
REPOSITORY TAG IMAGE ID CREATED SIZE
franck/demo latest 2ab439a723c4 32 minutes ago 214MB
alpine latest 3fd9065eaf02 2 months ago 4.15MB
 
[root@VM121 docker]# df -hT /var/lib/docker
Filesystem Type Size Used Avail Use% Mounted on
/dev/sdc xfs 80G 237M 80G 1% /var/lib/docker


multi-stage build

Finally, you can do something similar to an intermediate squash using multi-stage build.

Here is my Dockerfile:

FROM alpine:latest as staging
WORKDIR /var/tmp
ADD file0.100M .
RUN cp file0.100M file1.100M
RUN rm file0.100M
RUN mv file1.100M file2.100M
RUN dd if=/dev/urandom of=file2.100M seek=100 count=100 bs=1M
 
FROM alpine:latest
WORKDIR /var/tmp
COPY --from=staging /var/tmp .


With multi-stage build, we can start the second stage from a different image, and add more steps, but here I just start with the same alpine image and copy the final layer of the previous build.

We see something very similar to the –squash one:

[root@VM121 docker]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
franck/demo latest 55f329385f8c Less than a second ago 214MB
<none> <none> fd26a00db784 8 seconds ago 528MB
alpine latest 3fd9065eaf02 2 months ago 4.15MB
 
[root@VM121 docker]# docker image ls -a
REPOSITORY TAG IMAGE ID CREATED SIZE
franck/demo latest 55f329385f8c 1 second ago 214MB
<none> <none> fd26a00db784 9 seconds ago 528MB
<none> <none> 9bf5be367b63 32 seconds ago 319MB
<none> <none> 531d78833ba8 35 seconds ago 214MB
<none> <none> 05dd68114743 36 seconds ago 214MB
<none> <none> b9e5215a9fc8 39 seconds ago 109MB
<none> <none> ab332f486793 41 seconds ago 4.15MB
alpine latest 3fd9065eaf02 2 months ago 4.15MB

The history of the last stage shows the copy of 210MB from the previous one:

[root@VM121 docker]# docker image history franck/demo
IMAGE CREATED CREATED BY SIZE COMMENT
55f329385f8c 1 second ago /bin/sh -c #(nop) COPY dir:2b66b5c36eff5b51f… 210MB
ab332f486793 41 seconds ago /bin/sh -c #(nop) WORKDIR /var/tmp 0B
3fd9065eaf02 2 months ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B
<missing> 2 months ago /bin/sh -c #(nop) ADD file:093f0723fa46f6cdb… 4.15MB


The usage of filesystem is similar to the –squash one. Even if we reduced the final image, all the intermediate states had to be stored:

[root@VM121 docker]# df -hT /var/lib/docker
Filesystem Type Size Used Avail Use% Mounted on
/dev/sdc xfs 80G 737M 80G 1% /var/lib/docker


That looks good, if you accept to use a large intermediate space while building the image, which gives you the possibility to debug without re-running from the beginning, thanks to the layers in cache. However, you have still the inefficiency that each time you try the build, the context will be sent again even when not needed. And that is long with a 3GB .zip in the case of Oracle Database installation. Unfortunately, if you add the file to the .dockerignore once you know you have the ADD steps in cache, the next build attempt will not use the caches anymore. I would love to see a per-stage .dockerignore file for multi-stage builds. Or simply have docker realize that some files in the context will not be needed by the COPY or ADD that are not in cache yet.

Sending the whole context at each build attempt, when debugging your Dockerfile, is not efficient at all and looks like punch-card time compilation where people sent the cards to be compiled during the night. One syntax error on the first line and you go for another day.

One solution is to have all the required files in an NFS or HTTPd server and get them with ADD from the URL as mentioned earlier.

Multi-stage with multi-contexts

Another solution is to put all COPY or ADD from context in one Dockerfile to build the image containing all required files, and then build your image from it (and squash it at the end).

Here is my first Dockerfile, just adding the files from the context:

[root@VM121 docker]# ls /var/tmp/demo
Dockerfile file0.100M nocontext
[root@VM121 docker]# cat /var/tmp/demo/Dockerfile
FROM alpine:latest as staging
WORKDIR /var/tmp
ADD file0.100M .


I build this ‘staging’ image:

[root@VM121 docker]# docker image build -t franck/stage0 /var/tmp/demo
Sending build context to Docker daemon 104.9MB
Step 1/3 : FROM alpine:latest as staging
latest: Pulling from library/alpine
ff3a5c916c92: Pull complete
Digest: sha256:7df6db5aa61ae9480f52f0b3a06a140ab98d427f86d8d5de0bedab9b8df6b1c0
Status: Downloaded newer image for alpine:latest
---> 3fd9065eaf02
Step 2/3 : WORKDIR /var/tmp
Removing intermediate container 0eeed8e0cfd2
---> a5db3b29c8e1
Step 3/3 : ADD file0.100M .
---> 2a34e1e981be
Successfully built 2a34e1e981be
Successfully tagged franck/stage0:latest


This one is the minimal one:

[root@VM121 docker]# docker image ls
+ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
franck/stage0 latest 2a34e1e981be Less than a second ago 109MB
alpine latest 3fd9065eaf02 2 months ago 4.15MB
 
[root@VM121 docker]# df -hT /var/lib/docker
Filesystem Type Size Used Avail Use% Mounted on
/dev/sdc xfs 80G 139M 80G 1% /var/lib/docker


Now, I don’t need to send this context anymore during further development of my Dockerfile.
I’ve added the following steps to a Dockerfile in another directory:

[root@VM121 docker]# ls /var/tmp/demo/nocontext/
Dockerfile
[root@VM121 docker]# cat /var/tmp/demo/nocontext/Dockerfile
FROM franck/stage0 as stage1
WORKDIR /var/tmp
RUN cp file0.100M file1.100M
RUN rm file0.100M
RUN mv file1.100M file2.100M
RUN dd if=/dev/urandom of=file2.100M seek=100 count=100 bs=1M
FROM alpine:latest
WORKDIR /var/tmp


Here is the build, using multi-stage to get a squashed final image (you can also use –squash)

[root@VM121 docker]# docker image build -t franck/demo /var/tmp/demo/nocontext
 
Sending build context to Docker daemon 2.048kB
Step 1/9 : FROM franck/stage0 as stage1
---> 2a34e1e981be
Step 2/9 : WORKDIR /var/tmp
Removing intermediate container eabf57a8de05
...
Successfully built 82478bfa260d
Successfully tagged franck/demo:latest


At that point, there’s no advantage on space used as I keep all layers for easy Dockerfile development:

[root@VM121 docker]# df -hT /var/lib/docker
Filesystem Type Size Used Avail Use% Mounted on
/dev/sdc xfs 80G 738M 80G 1% /var/lib/docker
 
[root@VM121 docker]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
franck/demo latest 82478bfa260d About a minute ago 214MB
<none> <none> 5772ad68d208 About a minute ago 528MB
franck/stage0 latest 2a34e1e981be About a minute ago 109MB
alpine latest 3fd9065eaf02 2 months ago 4.15MB


But now, if I want to add an additional step:

[root@VM121 docker]# cat >> /var/tmp/demo/nocontext/Dockerfile <<< 'RUN chmod a+x /var/tmp'


I can re-build quickly, using cached layers, and without the need to send the context again:

[root@VM121 docker]# docker image build -t franck/demo /var/tmp/demo/nocontext
Sending build context to Docker daemon 2.048kB
Step 1/10 : FROM franck/stage0 as stage1
---> 2a34e1e981be
Step 2/10 : WORKDIR /var/tmp
---> Using cache
---> fa562926cc2b
Step 3/10 : RUN cp file0.100M file1.100M
---> Using cache
---> 31ac716f4d61
Step 4/10 : RUN rm file0.100M
---> Using cache
---> d7392cf51ad9
Step 5/10 : RUN mv file1.100M file2.100M
---> Using cache
---> 4854e503885b
Step 6/10 : RUN dd if=/dev/urandom of=file2.100M seek=100 count=100 bs=1M
---> Using cache
---> 5772ad68d208
Step 7/10 : FROM alpine:latest
---> 3fd9065eaf02
Step 8/10 : WORKDIR /var/tmp
---> Using cache
---> a5db3b29c8e1
Step 9/10 : COPY --from=stage1 /var/tmp .
---> Using cache
---> 82478bfa260d
Step 10/10 : RUN chmod a+x /var/tmp
---> 4a69ee40a938
Successfully built 4a69ee40a938
Successfully tagged franck/demo:latest


Once I’m ok with my final image, I can remove the intermediate ones:

[root@VM121 docker]# docker image prune -f
Deleted Images:
deleted: sha256:5772ad68d20841197d1424f7c64edd21704e4c7b470acb2193de51ae8741385d
deleted: sha256:bab572d749684d126625a74be4f01cc738742f9c112a940391e3533e61dd55b9
deleted: sha256:4854e503885b4057809fe2867a743ae7898e3e06b329229519fdb5c9d8b10ac1
deleted: sha256:de4acb90433c30d6a21cc3b4483adbd403d8051f3c7c31e6bc095a304606355a
deleted: sha256:d7392cf51ad99d5d0b7a1a18d8136905c87bc738a5bc94dec03e92f5385bf9c8
deleted: sha256:f037e7f973f4265099402534cd7ba409f35272701166d59a1be8e5e39508b07c
deleted: sha256:31ac716f4d61f0048a75b8de6f18757970cf0670a0a3d711e4386bf098b32041
deleted: sha256:2dccb363c5beb4daf45586383df6454b198f824d52676f70318444c346c0fe9a
deleted: sha256:fa562926cc2b3cb56400e1068984bb4048f56713a3cf6dcfa3cf6d945023ebc4
 
Total reclaimed space: 419.4MB

And the staging one:

[root@VM121 docker]# docker image rm franck/stage0
Untagged: franck/stage0:latest
Deleted: sha256:2a34e1e981be9154c31c5ee7eb942cc121267f4416b6fe502ab93f2dceafd98c
Deleted: sha256:b996a1bdc829167f16dcbe58b717284764470661c3116a6352f15012e1dff07c


Finally, I optimized the developement of the Dockerfile and finished with the minimal size.

[root@VM121 docker]# df -hT /var/lib/docker
Filesystem Type Size Used Avail Use% Mounted on
/dev/sdc xfs 80G 237M 80G 1% /var/lib/docker


So what?

I’m always surprised by the lack of efficiency when building an image with a Dockerfile. Any serious application deployment involves several intermediate files and the way docker build is layered inflates the size and the time required. Efficient layering and snapshotting work at block level. Here, at file level, any byte of data modified in a file, even metadata such as the file name, is a whole file copy. But for common applications, the installation steps are not as simple adding new files. You may have files appended, object files added to libraries, then compiled, the stripped…

In this post, I tested some recent features, such as multi-stage build and the experimental –squash, as well as a simple manual multi-stage build. Of course, you can do everything in the same layers, and even not use Dockerfiles at all, but then why using Docker? There’s also the Packer approach that I’ve not tested yet. However, I like the Docker approach, but only when used correctly. Deploying an application, like Oracle Database, should use the layered build in the following way: additional steps for new options or new updates. This means that the files must be built elsewhere, in a staging container, and added in one step. And to be efficient, the context should be sent only when needed: when a non-cached ADD or COPY requires it.

Cet article Docker: efficiently building images for large software est apparu en premier sur Blog dbi services.

At each new Oracle version, I like to check what’s new, not only from the documentation, but also from exposed internals. I look (and sometimes diff) on catalog views definitions, undocumented parameters, and even the new C functions in the libraries. At last Oak Table World, I was intrigued by this V$SQLFN_METADATA view explained by Vit Spinka when digging into the internals of how execution plans are stored. This view has entries with all SQL functions, and a VERSION column going from ‘V6 Oracle’ to ‘V11R1 Oracle’. The lastest functions has an ‘INVALID’ entry and we also can see some functions with ‘SQL/DS’. Well, now that we have Oracle 18c on the Oracle Cloud, I came back to this view to see if anything is new, listing the highest FUNC_ID at the top and the first row attired my attention:


SQL> select * from V$SQLFN_METADATA order by 1 desc fetch first 10 rows only;
 
FUNC_ID NAME MINARGS MAXARGS DATATYPE VERSION ANALYTIC AGGREGATE OFFLOADABLE DISP_TYPE USAGE DESCR CON_ID
------- ---- ------- ------- -------- ---------- -------- --------- ----------- --------- ----- ----- ------
1148 TO_DOG_YEAR 1 4 NUMERIC V13 Oracle NO NO YES NORMAL TO_DOG_YEAR 0
1147 JSON_MERGEPATCH 4 0 UNKNOWN INVALID NO NO NO NORMAL JSON_MERGEPATCH 0
1146 JSON_PATCH 4 0 UNKNOWN INVALID NO NO NO NORMAL JSON_PATCH 0
1145 ROUND_TIES_TO_EVEN 1 2 NUMERIC INVALID NO NO YES NORMAL ROUND_TIES_TO_EVEN 0
1144 CON_ID_TO_CON_NAME 1 0 UNKNOWN INVALID NO NO NO NORMAL CON_ID_TO_CON_NAME 0
1143 TIMESTAMP_TO_NUMBER 1 1 UNKNOWN INVALID NO NO YES NORMAL TIMESTAMP_TO_NUMBER 0
1142 TO_UTC_TIMESTAMP_TZ 1 0 UNKNOWN INVALID NO NO YES NORMAL TO_UTC_TIMESTAMP_TZ 0
1141 OPTSYSAPPROXRANK 1 0 UNKNOWN INVALID NO NO NO NORMAL Internal evaluation function for multiple approx_rank's 0
1140 APPROX_RANK 1 1 NUMERIC INVALID NO YES NO NORMAL APPROX_RANK 0
1139 APPROX_SUM 1 2 NUMERIC INVALID NO YES NO NORMAL APPROX_SUM 0


Because those functions are SQL functions, I searched this ‘TO_DOG_YEAR’ on Google to see whether a new ANSI SQL function was implemented. But finally came upon something I didn’t expect: Dog Years Calculator. The trends in databases are really going crazy these times. All focus is on developers. XML, JSON, Docker… and now a function to calculate your age in dog years.
But afterall, it makes sense. IoT (not ‘Index Organized Table’ but ‘Internet Of Things’) is coming with sensors everywhere. And it is not only ‘things’ but it comes to living beings. I have read recently about ‘Internet of Pets’ where collars equipped with sensors detect where your domestic animal go and when he is hungry.

Let’s test it. Tomorrow, my elder kid has his 13th birthday. Now Oracle can tell me that he will be 65 in dog years:

SQL> select to_dog_year(date'2005-04-02') from dual;
 
TO_DOG_YEAR(DATE'2005-04-02')
-----------------------------
65

Yes, here I learn that the calculation is a bit more complex than just multiplying by 7. Of course, adding a SQL standard function would not make sense if it was just a multiplication.

But it seems to be even more complex. I searched for the C functions behind this one:

[oracle@CLOUD18C ~]$ nm /u01/app/oracle/product/18.0.0/dbhome_1/bin/oracle | grep -iE "dog.*year"
000000001452e073 r KNCLG_TODOGYEAR
0000000003ffcf40 T LdiJDaysDogYear
000000000f3170c0 T LdiJulianDogYear
000000000f316fc0 T LdiJulianDogYeararr
000000000f3170f0 t LdiJulianDogYeari
000000000f606e10 T OCIPConvertDateDogYearTime
000000000ebf2380 t qerxjConvertDogYearTime
0000000010de19e0 t qjsngConvStructDogYear
0000000010de0320 T qjsngNumberDogYearDty
0000000010de06f0 T sageStringDogYearDty
0000000010de7110 T sageplsDogYear
000000000bc5cd80 t sagerwAddDogYearTime
0000000010bad3c0 T qmxtgrConvSaxDogYear
0000000010bad400 T qmxtgrConvSaxDogYear_internal
00000000025ae090 T qosDateTimeDogYear
0000000004f22b60 T xsCHDogYeartime
000000000438c230 T nlsBreedDogYear
000000000438bb50 t nlsBreedDogYearCmn
000000000438c060 T nlsBreedDogYearTime
000000000438bc50 T nlsBreedDogYear
00000000044d1da0 T xvopAddDTDurDogYear
00000000044d1ac0 T xvopAddYMDurDogYear


Those ‘nlsBreed’ functions ring a bell and I checked if there are new values in V$NLS_VALID_VALUES

SQL> select distinct parameter from V$NLS_VALID_VALUES;
 
PARAMETER
----------------------------------------------------------------
TERRITORY
CHARACTERSET
BREED
LANGUAGE
SORT


That ‘BREED’ is a new one, with a lot of interesting values:

And here is my example using this new NLS parameter.

SQL> select to_dog_year(date'2005-04-02','','NLS_BREED=Saint Bernard') from dual;
 
TO_DOG_YEAR(DATE'2005-04-02',
----------------------------
96


Note that I’ve no idea about the second parameter, I had to put a ‘null’ for it to be able to mention the NLS one, or I got a ‘ORA-00909: invalid number of arguments’.

I have to say that, for a DBA focused on the core database functions, it is hard to understand that new features go on things like this TO_DOG_YEAR function. But being realistic, it is clear that the budget for new features go into the new direction: all for developers, big data, IoT… Of course we can write those functions in PL/SQL or maybe one day with JavaScript thanks to the Multi-Lingual Engine currently in beta. But IoT is also about performance, and a standard function avoids context switches.

Added a few hours later

There are a lot of questions about this new function. Here are some links to go further as many people in the Oracle Community have analyzed it further:

Martin Berger tested performance: http://berxblog.blogspot.ch/2018/04/more-fun-with-ages.html
Pieter Van Puymbroeck realized it was offloaded in Exadata: http://vanpupi.stepi.net/2018/04/01/exploring-18c-exadata-functions/
Brendan Thierney reveald a project he worked on in beta: http://www.oralytics.com/2018/04/predicting-ibs-in-dogs-using-oracle-18c.html
Øyvind Isene provides a way to test it with a cloud discount: http://oisene.blogspot.ch/2018/04/oracle-is-best-database-for-your-pets.html

Update 2-APR-2018

A little update for those who didn’t realize this was posted on 1st of April. It was an April Fool common idea from some Oracle Community buddies on the post-UKOUG_TECH17 trip. And what remains true all the year is how this community is full of awesome people. And special thanks to Connor who added great ideas here

Cet article After IoT, IoP makes its way to the database est apparu en premier sur Blog dbi services.

Dbvisit Standby version 8 comes with a nice feature, a daemon, which gives the benefit to send and apply the archive log automatically in the background. Bypassing the system scheduling, the daemon will facilitate customer RPO (Recovery Point Objective) and RTO (Recovery Time Objective) fine tuning. Monitoring to apply logs to the Standby only when needed, will also optimize use of resources. Originally available for the Linux based environments, the feature has been made available for the Windows based platforms starting 8.0.06. This blog will cover its implementation and show its benefit.

Demo databases environments have been easily managed thanks to DBI DMK tool.

Environment

DBVP : Primary Server

DBVS : Standby Server

DBVPDB_SITE1 : Primary database

DBVPDB_SITE2 : Physical Standby database

Daemon start/stop/status

oracle@DBVP:/home/oracle/ [DBVPDB] /u01/app/dbvisit/standby/dbvctl -d DBVPDB_SITE1 -D start
Starting Dbvisit Daemon...
Started successfully.

oracle@DBVP:/home/oracle/ [DBVPDB] /u01/app/dbvisit/standby/dbvctl -d DBVPDB_SITE1 -D status
Dbvisit Daemon process is running with pid 11546.

oracle@DBVP:/home/oracle/ [DBVPDB] /u01/app/dbvisit/standby/dbvctl -d DBVPDB_SITE1 -D stop
Stopping Dbvisit Daemon...
Successfully stopped.

Automatic startup

In order to start the daemon automatically at boot,  and easily manage its status, we will create a dbvlogdaemon Service.

[root@DBVP ~]# vi /etc/systemd/system/dbvlogdaemon.service

[root@DBVP ~]# cat /etc/systemd/system/dbvlogdaemon.service
[Unit]
Description=DB Visit log daemon Service
After=oracle.service

[Service]
Type=simple
RemainAfterExit=yes
User=oracle
Group=oinstall
Restart=always
ExecStart=/u01/app/dbvisit/standby/dbvctl -d DBVPDB_SITE1 -D start
ExecStop=/u01/app/dbvisit/standby/dbvctl -d DBVPDB_SITE1 -D stop

[Install]
WantedBy=multi-user.target

[root@DBVP ~]# chmod 644 /etc/systemd/system/dbvlogdaemon.service

[root@DBVP ~]# systemctl daemon-reload

[root@DBVP ~]# systemctl enable dbvlogdaemon.service

Of course this would not avoid impact in case of daemon crash which could be simulated with a kill command.

Check running daemon

oracle@DBVP:/u01/app/dbvisit/standby/ [DBVPDB] ps -ef | grep dbvctl | grep -v grep
oracle    4299     1  0 08:25 ?        00:00:02 /u01/app/dbvisit/standby/dbvctl -d DBVPDB_SITE1 -D start

oracle@DBVP:/u01/app/dbvisit/standby/ [DBVPDB] ./dbvctl -d DBVPDB_SITE1 -D status
Dbvisit Daemon process is running with pid 4299.

oracle@DBVS:/u01/app/dbvisit/standby/ [DBVPDB] ps -ef | grep dbvctl | grep -v grep
oracle    4138     1  0 08:25 ?        00:00:01 /u01/app/dbvisit/standby/dbvctl -d DBVPDB_SITE1 -D start

oracle@DBVS:/u01/app/dbvisit/standby/ [DBVPDB] ./dbvctl -d DBVPDB_SITE1 -D status
Dbvisit Daemon process is running with pid 4138.

Daemon Parameter

# DMN_DBVISIT_INTERVAL     - interval in sec for dbvisit schedule on source
# DMN_MONITOR_INTERVAL     - interval in sec for log monitor schedule on source
# DMN_DBVISIT_TIMEOUT      - max sec for a dbvisit process to complete on source
# DMN_MONITOR_TIMEOUT      - max sec for a monitor process to complete on source
# DMN_MONITOR_LOG_NUM      - number of logs to monitor on source
# DMN_MAX_FAIL_NOTIFICATIONS - max number of emails sent on failure on source
# DMN_BLACKOUT_STARTTIME   - blackout window start time HH:MI on source
# DMN_BLACKOUT_ENDTIME     - blackout window end time HH:MI on source
# DMN_DBVISIT_INTERVAL_DR  - interval in sec for dbvisit schedule on destination
# DMN_MONITOR_INTERVAL_DR  - interval in sec for log monitor schedule on destination
# DMN_DBVISIT_TIMEOUT_DR   - max sec for a dbvisit process to complete on destination
# DMN_MONITOR_TIMEOUT_DR   - max sec for a monitor process to complete on destination
# DMN_MONITOR_LOG_NUM_DR   - number of logs to monitor on destination
# DMN_MAX_FAIL_NOTIFICATIONS_DR - max number of emails sent on failure on destination
# DMN_BLACKOUT_STARTTIME_DR- blackout window start time HH:MI on destination
# DMN_BLACKOUT_ENDTIME_DR  - blackout window end time HH:MI on destination

With the daemon, we can pause the archive send/apply process using the DMN_BLACKOUT parameters.

To setup our lab we will act on the most important parameters :

• DMN_MONITOR_INTERVAL  (Primary) and DMN_MONITOR_INTERVAL_DR (Standby).
  The Monitor Interval will give the frequency for Dbvisit to check for new archive log and only act if existing.
• DMN_DBVISIT_INTERVAL (Primary) and DMN_DBVISIT_INTERVAL_DR (Standby)
  The Dbvisit Interval will give the frequency for Dbvisit to force a send/apply process. This action will be dependent of the LOGSWITCH DCC parameter. Recommendation is not to go below 5 minutes.

oracle@DBVP:/oracle/u01/app/dbvisit/standby/conf/ [DBVPDB] pwd
/oracle/u01/app/dbvisit/standby/conf

oracle@DBVP:/u01/app/dbvisit/standby/conf/ [DBVPDB] egrep 'DMN_DBVISIT_INTERVAL|DMN_MONITOR_INTERVAL' dbv_DBVPDB_SITE1.env
# DMN_DBVISIT_INTERVAL     - interval in sec for dbvisit schedule on source
# DMN_MONITOR_INTERVAL     - interval in sec for log monitor schedule on source
# DMN_DBVISIT_INTERVAL_DR  - interval in sec for dbvisit schedule on destination
# DMN_MONITOR_INTERVAL_DR  - interval in sec for log monitor schedule on destination
DMN_DBVISIT_INTERVAL = 300
DMN_MONITOR_INTERVAL = 60
DMN_DBVISIT_INTERVAL_DR = 300
DMN_MONITOR_INTERVAL_DR = 60

The LOGSWITCH parameter determines if a database log switch (alter system switch logfile) should be trigger at Dbvisit execution.
N (default value) : Only if there are no new archive logs to transfer.
Y : At every execution, independently of the archive log creation.
I(Ignore) : Never. To be use with caution.

A daemon restart is mandatory post DDC configuration file updates.

[root@DBVP ~]# service dbvlogdaemon stop
Redirecting to /bin/systemctl stop dbvlogdaemon.service
[root@DBVP ~]# service dbvlogdaemon start
Redirecting to /bin/systemctl start dbvlogdaemon.service

[root@DBVS ~]# service dbvlogdaemon stop
Redirecting to /bin/systemctl stop dbvlogdaemon.service
[root@DBVS ~]# service dbvlogdaemon start
Redirecting to /bin/systemctl start dbvlogdaemon.service

Send and apply archive log demo

Get current date and primary current sequence.

SQL> select sysdate from dual;

SYSDATE
-------------------
2018/03/28 12:30:50

SQL> select max(sequence#) from v$log;

MAX(SEQUENCE#)
--------------
           179

Generate a Dbvisit gap report.

oracle@DBVP:/u01/app/dbvisit/standby/conf/ [DBVPDB] /u01/app/dbvisit/standby/dbvctl -d DBVPDB_SITE1 -i
=============================================================
Dbvisit Standby Database Technology (8.0.16_0_g4e0697e2) (pid 21393)
dbvctl started on DBVP: Wed Mar 28 12:30:57 2018
=============================================================

Dbvisit Standby log gap report for DBVPDB_SITE1 thread 1 at 201803281230:
-------------------------------------------------------------
Destination database on DBVS is at sequence: 178.
Source database on DBVP is at log sequence: 179.
Source database on DBVP is at archived log sequence: 178.
Dbvisit Standby last transfer log sequence: 178.
Dbvisit Standby last transfer at: 2018-03-28 12:29:14.

Archive log gap for thread 1:  0.
Transfer log gap for thread 1: 0.
Standby database time lag (DAYS-HH:MI:SS): +00:01:27.


=============================================================
dbvctl ended on DBVP: Wed Mar 28 12:31:06 2018
=============================================================

No archive log needs to be send and apply on the standby. Both databases are in sync.

Generate logfile switch

SQL> alter system switch logfile;

System altered.

SQL> alter system switch logfile;

System altered.

SQL> alter system switch logfile;

System altered.

Check current date and primary database current sequence.

SQL> select sysdate from dual;

SYSDATE
-------------------
2018/03/28 12:31:29

SQL> select max(sequence#) from v$log;

MAX(SEQUENCE#)
--------------
           182

Generate new Dbvisit gap reports.

oracle@DBVP:/u01/app/dbvisit/standby/conf/ [DBVPDB] /u01/app/dbvisit/standby/dbvctl -d DBVPDB_SITE1 -i
=============================================================
Dbvisit Standby Database Technology (8.0.16_0_g4e0697e2) (pid 21454)
dbvctl started on DBVP: Wed Mar 28 12:31:38 2018
=============================================================

Dbvisit Standby log gap report for DBVPDB_SITE1 thread 1 at 201803281231:
-------------------------------------------------------------
Destination database on DBVS is at sequence: 178.
Source database on DBVP is at log sequence: 182.
Source database on DBVP is at archived log sequence: 181.
Dbvisit Standby last transfer log sequence: 178.
Dbvisit Standby last transfer at: 2018-03-28 12:29:14.

Archive log gap for thread 1:  3.
Transfer log gap for thread 1: 3.
Standby database time lag (DAYS-HH:MI:SS): +00:02:27.


=============================================================
dbvctl ended on DBVP: Wed Mar 28 12:31:47 2018
=============================================================

We can see that we have 3 new archive logs to transfer and to apply on the standby.
There is a 3 sequences lag between both databases.

oracle@DBVP:/u01/app/dbvisit/standby/conf/ [DBVPDB] /u01/app/dbvisit/standby/dbvctl -d DBVPDB_SITE1 -i
=============================================================
Dbvisit Standby Database Technology (8.0.16_0_g4e0697e2) (pid 21571)
dbvctl started on DBVP: Wed Mar 28 12:32:19 2018
=============================================================
Dbvisit Standby log gap report for DBVPDB_SITE1 thread 1 at 201803281232:
-------------------------------------------------------------
Destination database on DBVS is at sequence: 178.
Source database on DBVP is at log sequence: 182.
Source database on DBVP is at archived log sequence: 181.
Dbvisit Standby last transfer log sequence: 181.
Dbvisit Standby last transfer at: 2018-03-28 12:32:13.
Archive log gap for thread 1:  3.
Transfer log gap for thread 1: 0.
Standby database time lag (DAYS-HH:MI:SS): +00:02:27.
=============================================================
dbvctl ended on DBVP: Wed Mar 28 12:32:27 2018
=============================================================

3 archive logs has been automatically transferred by the daemon to the standby in the next minute.

oracle@DBVP:/u01/app/dbvisit/standby/conf/ [DBVPDB] /u01/app/dbvisit/standby/dbvctl -d DBVPDB_SITE1 -i
=============================================================
Dbvisit Standby Database Technology (8.0.16_0_g4e0697e2) (pid 21679)
dbvctl started on DBVP: Wed Mar 28 12:33:00 2018
=============================================================

Dbvisit Standby log gap report for DBVPDB_SITE1 thread 1 at 201803281233:
-------------------------------------------------------------
Destination database on DBVS is at sequence: 181.
Source database on DBVP is at log sequence: 182.
Source database on DBVP is at archived log sequence: 181.
Dbvisit Standby last transfer log sequence: 181.
Dbvisit Standby last transfer at: 2018-03-28 12:32:13.

Archive log gap for thread 1:  0.
Transfer log gap for thread 1: 0.
Standby database time lag (DAYS-HH:MI:SS): +00:01:13.


=============================================================
dbvctl ended on DBVP: Wed Mar 28 12:33:09 2018
=============================================================

Another minute later the standby daemon applied the new archive logs. Both databases are on sync.

Conclusion

Dbvisit new daemon feature is adding real flexibility in sending and applying archive logs, and help improving customer RPO and RTO. We still might want to keep a daily crontab gap report with email to be sent to a DBA team. This will ensure to monitor daemon keep alive.

Logswitch and sending archive logs to standby consumes real system resource. Dbvisit daemon will also help fine tuning the use of the resource.

Note that the daemon processes must be restarted after each daylight saving clock change, and stopped during graceful switchover.

Cet article Dbvisit Standby Archive Log Daemon est apparu en premier sur Blog dbi services.

A covering index is an index that contains all the columns required by your query, so that you don’t have to do a TABLE ACCESS BY INDEX ROWID, which is the major cost of an index range scan. You don’t need any special feature to do that in Oracle. Just add the required columns at the end of the index. In the execution plan you will see the columns used as index keys for the range scan displayed in ‘access’ predicates, and the further filtering done on the remaining columns with ‘filter’ predicates. The ‘projection’ shows the columns that are returned in the rowset result.
However you may have seen that SQL Server has a special ‘INCLUDING’ keyword to separate those non-key columns added only for filtering or projection but not for access. What does it bring that Oracle doesn’t have?

An index entry is composed of a key and data associated to the key. The index is sorted on the key. The data for each key have no special order, like in a heap table. The idea of the SQL Server INCLUDING keyword is to separate the columns belonging to the key and the columns belonging to the data. It is not mandatory. You can add all columns to the key but depending on the implementation, the benefit can be:

• some data types may not be allowed in the key but allowed as data
• sorting the data when not required may be a performance overhead
• there can be limitations on the size of the key
• having a larger key may require more space in the branches
• adding sorted columns may change the clustering factor

In Oracle, there are very few data types that cannot be indexed (like LONG). The limitation on the size of the key may come into play for large 12c Extended Datatypes. You can substring them, but that defeats the goal of covering indexes. I see two reasons why ‘INCLUDING’ indexes can be useful. The first reason is about the clustering factor. The second about sorting the whole index entry and referencing it from the branches. I’ll detail those reasons later, but first here is an example.


SQL> create table DEMO (UNIQU ,RANGE ,RANDOM_TEXT ,CONSTANT_TEXT ) as select rownum UNIQU , mod(rownum,4) RANGE , dbms_random.string('u',80) RANDOM_TEXT , lpad('x',80,'x') CONSTANT_TEXT from xmltable('1 to 100000');
Table DEMO created.
SQL> commit;
Commit complete.

This table has an all-distinct-values column UNIQ, a few-distinct-values on (RANGE) and I’ll use them for the key. And I’ve two columns I’ll add as additional column for covering queries: one is with lot of distinct values (RANDOM_TEXT) and the other has few distinct values (CONSTANT_TEXT).
The first rows look like this:

SQL> select * from DEMO order by ROWID fetch first 5 rows only;
UNIQU RANGE RANDOM_TEXT CONSTANT_TEXT
----- ----- -------------------------------------------------------------------------------- --------------------------------------------------------------------------------
1 1 XCFNWCRCFBEPJPSHREUVVVTBUCCXLZMRPJPNQDTHWYRZRUORBPDOBCIRFHLICETULTCZTMPOCMUNQITV xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2 2 XUSPNDOMPQKOIRCVDDTVYAGKRDGIXOSVUNMRAQLSRQGYKOFEXRQMCPXPYZYKRHHKDXGIINOUUAUJOLOO xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
3 3 ZBCVFTDSRUFIUTSIWOOOBWIRMEFUXNWLADAPUPFNPVYDLPQTOUZVXJKMGIPCGZESXFXOIYVMKNSMMZKB xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
4 0 VOIRCXFVSRVZQRZDRLQRHZWNGQJAAWJXWXJKRCJVPWYDJSZLJIOEWAMCFSRCUPSPPEKITJYHHOUQSVYQ xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
5 1 UUSAMEVRWNLPGCUVMJWVVPDAENRYKIWWMIHTUJSZRQASMTYOVQNCGZGZIJZWNSOJVSIBMMUEAXOHJCOA xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx


I’m adding indexes fo access on RANGE as the index key, with only the key, or covering the random or constant text:

SQL> create index DEMO_RANGE on DEMO(RANGE) pctfree 50;
Index DEMO_RANGE created.
SQL> create index DEMO_RANGE_COVERING_RANDOM on DEMO(RANGE,RANDOM_TEXT) pctfree 50;
Index DEMO_RANGE_COVERING_RANDOM created.
SQL> create index DEMO_RANGE_COVERING_CONSTANT on DEMO(RANGE,CONSTANT_TEXT) pctfree 50;
Index DEMO_RANGE_COVERING_CONSTANT created.

An additional one adding the unique column in-between:

SQL> create index DEMO_RANGE_COVERING_WITH_PK on DEMO(RANGE,UNIQU,CONSTANT_TEXT) pctfree 50;
Index DEMO_RANGE_COVERING_WITH_PK created.

And now for access with the unique column as a key:

SQL> create index DEMO_UNIQU_COVERING_RANDOM on DEMO(UNIQU,RANDOM_TEXT) pctfree 50;
Index DEMO_UNIQU_COVERING_RANDOM created.
SQL> create index DEMO_UNIQU_COVERING_CONSTANT on DEMO(UNIQU,CONSTANT_TEXT) pctfree 50;
Index DEMO_UNIQU_COVERING_CONSTANT created.


Here are some interesting stats:

SQL> exec dbms_stats.gather_table_stats(user,'DEMO');
PL/SQL procedure successfully completed.
SQL> select index_name,blevel,leaf_blocks,num_rows,clustering_factor from user_indexes where table_name='DEMO' order by 2,3;
INDEX_NAME BLEVEL LEAF_BLOCKS NUM_ROWS CLUSTERING_FACTOR
-------------------------------- ------ ----------- -------- -----------------
DEMO_RANGE 1 353 100000 9757
DEMO_RANGE_COVERING_RANDOM 2 2440 100000 99967
DEMO_RANGE_COVERING_CONSTANT 2 2440 100000 9757
DEMO_UNIQU_COVERING_RANDOM 2 2500 100000 2440
DEMO_UNIQU_COVERING_CONSTANT 2 2500 100000 2440
DEMO_RANGE_COVERING_WITH_PK 2 2565 100000 9757
6 rows selected.


Leaf size

About the size, the covering indexes have approximately the same number of leaf blocks because the included column (RANDOM_TEXT or CONSTANT_TEXT) has the same size (80 bytes). Of course, the non-covering index is smaller (but will need table access to query additional column). The key on UNIQU is slightly larger than the one on RANGE because the numbers go higher. The index with 3 columns is the largest.

Clustering factor

About the clustering factor, there’s one outlier here which deserves an explanation. But before that, you must understand that this higher clustering factor is not important for a query using the covering index, such as a SELECT RANDOM_TEXT WHERE RANGE=0, because in that case you don’t read the table. However for some queries you may cover only the filter predicates and go to the table for projection.
But the big problem is that when you add a column to an index to address a specific query, you don’t want to risk a side effect on another query, and changing the clustering factor is a risk here. One solution is to keep the old non-covering index (DEMO_RANGE) but then the side effect is on DML overhead.

To understand the change in clustering factor we must go deeper on Oracle index key and data implementation. The ‘data’ part exists in Oracle indexes even when not specified explicitely with an INCLUDING clause. The ROWID is the data part. An index entry associates a key (the indexed columns) with a pointer to the table row (the ROWID). At least, this is for UNIQUE indexes where each key is unique.

Non-unique indexes are a special case. Actually, Oracle implements only unique key indexes. When the indexed columns are not unique, the ROWID is stored on the key part of the index entry, and there is no data part. You should read Richard Foote, Differences between Unique and Non-Unique Indexes for detailed explanation.

Branch size

The previous statistics displayed only the number of branch level, which was the same, but we can have more detail about the branch size with an ANALYZE INDEX.

The non-covering index has only one branch block, the root, which references all the 353 leaf blocks containing the 100000 entries, with an average of 5479/352=15 bytes per branch entry:

SQL> analyze index DEMO_RANGE validate structure offline;
Index DEMO_RANGE analyzed.
SQL> select height,blocks,lf_blks,lf_rows_len,br_rows,br_blks,br_rows_len,most_repeated_key,btree_space,used_space,pct_used,rows_per_key,blks_gets_per_access,opt_cmpr_pctsave,lf_uncmp_rows_len,lf_uncmp_blks from index_stats
HEIGHT BLOCKS LF_BLKS LF_ROWS_LEN BR_ROWS BR_BLKS BR_ROWS_LEN MOST_REPEATED_KEY BTREE_SPACE USED_SPACE PCT_USED ROWS_PER_KEY BLKS_GETS_PER_ACCESS OPT_CMPR_PCTSAVE LF_UNCMP_ROWS_LEN LF_UNCMP_BLKS
------ ------ ------- ----------- ------- ------- ----------- ----------------- ----------- ---------- -------- ------------ -------------------- ---------------- ----------------- -------------
2 384 353 1375000 352 1 5479 25000 2830616 1380479 49 25000 12502.5 19 1375000 353


The covering index with lot of distinct values for the non-key columns has more branch blocks, with an average of 34623/2439=14 bytes per branch entry:

SQL> analyze index DEMO_RANGE_COVERING_RANDOM validate structure offline;
Index DEMO_RANGE_COVERING_RANDOM analyzed.
SQL> select height,blocks,lf_blks,lf_rows_len,br_rows,br_blks,br_rows_len,most_repeated_key,btree_space,used_space,pct_used,rows_per_key,blks_gets_per_access,opt_cmpr_pctsave,lf_uncmp_rows_len,lf_uncmp_blks from index_stats
HEIGHT BLOCKS LF_BLKS LF_ROWS_LEN BR_ROWS BR_BLKS BR_ROWS_LEN MOST_REPEATED_KEY BTREE_SPACE USED_SPACE PCT_USED ROWS_PER_KEY BLKS_GETS_PER_ACCESS OPT_CMPR_PCTSAVE LF_UNCMP_ROWS_LEN LF_UNCMP_BLKS
------ ------ ------- ----------- ------- ------- ----------- ----------------- ----------- ---------- -------- ------------ -------------------- ---------------- ----------------- -------------
3 2560 2440 9475000 2439 6 34623 1 19558408 9509623 49 1 4 2 9475000 2440

Here the number of branches is higher only because there are more leaves (as we have more columns), but not because of the size in the branch entries, which are even smaller. They are smaller because the branch does not have to store the full value of all columns in order to identify one leaf block. Then, only the first bytes are needed and not the full 80 bytes of them.

The covering index with few of distinct values for the non-key columns has a lot more branch blocks, with an average of 234755/2439=96 bytes per branch entry:

SQL> analyze index DEMO_RANGE_COVERING_CONSTANT validate structure offline;
Index DEMO_RANGE_COVERING_CONSTANT analyzed.
SQL> select height,blocks,lf_blks,lf_rows_len,br_rows,br_blks,br_rows_len,most_repeated_key,btree_space,used_space,pct_used,rows_per_key,blks_gets_per_access,opt_cmpr_pctsave,lf_uncmp_rows_len,lf_uncmp_blks from index_stats
 
HEIGHT BLOCKS LF_BLKS LF_ROWS_LEN BR_ROWS BR_BLKS BR_ROWS_LEN MOST_REPEATED_KEY BTREE_SPACE USED_SPACE PCT_USED ROWS_PER_KEY BLKS_GETS_PER_ACCESS OPT_CMPR_PCTSAVE LF_UNCMP_ROWS_LEN LF_UNCMP_BLKS
------ ------ ------- ----------- ------- ------- ----------- ----------------- ----------- ---------- -------- ------------ -------------------- ---------------- ----------------- -------------
3 2560 2440 9475000 2439 31 234755 25000 19759108 9709755 50 25000 12503.5 86 9475000 2440

So, here the size of the branch blocks is higher because we have multiple leaves blocks with the value of COVERING_CONSTANT the second column is not sufficient to identify only one leaf block. The full 80 bytes must be stored, and the rowid in addition to it.

When the indexed column has only unique values, there is no need to store more in the branches (not the additional columns, not the rowid) and only 12 bytes are needed here on average:

SQL> analyze index DEMO_UNIQU_COVERING_RANDOM validate structure offline;
Index DEMO_UNIQU_COVERING_RANDOM analyzed.
SQL> select height,blocks,lf_blks,lf_rows_len,br_rows,br_blks,br_rows_len,most_repeated_key,btree_space,used_space,pct_used,rows_per_key,blks_gets_per_access,opt_cmpr_pctsave,lf_uncmp_rows_len,lf_uncmp_blks from index_stats
 
HEIGHT BLOCKS LF_BLKS LF_ROWS_LEN BR_ROWS BR_BLKS BR_ROWS_LEN MOST_REPEATED_KEY BTREE_SPACE USED_SPACE PCT_USED ROWS_PER_KEY BLKS_GETS_PER_ACCESS OPT_CMPR_PCTSAVE LF_UNCMP_ROWS_LEN LF_UNCMP_BLKS
------ ------ ------- ----------- ------- ------- ----------- ----------------- ----------- ---------- -------- ------------ -------------------- ---------------- ----------------- -------------
3 2560 2500 9688892 2499 5 29737 1 20030140 9718629 49 1 4 0 9688892 2500


As the second column is not needed, the size of branch is the same whether we use RANDOM_TEXT or CONSTANT_TEXT:

SQL> analyze index DEMO_UNIQU_COVERING_CONSTANT validate structure offline;
Index DEMO_UNIQU_COVERING_CONSTANT analyzed.
SQL> select height,blocks,lf_blks,lf_rows_len,br_rows,br_blks,br_rows_len,most_repeated_key,btree_space,used_space,pct_used,rows_per_key,blks_gets_per_access,opt_cmpr_pctsave,lf_uncmp_rows_len,lf_uncmp_blks from index_stats
 
HEIGHT BLOCKS LF_BLKS LF_ROWS_LEN BR_ROWS BR_BLKS BR_ROWS_LEN MOST_REPEATED_KEY BTREE_SPACE USED_SPACE PCT_USED ROWS_PER_KEY BLKS_GETS_PER_ACCESS OPT_CMPR_PCTSAVE LF_UNCMP_ROWS_LEN LF_UNCMP_BLKS
------ ------ ------- ----------- ------- ------- ----------- ----------------- ----------- ---------- -------- ------------ -------------------- ---------------- ----------------- -------------
3 2560 2500 9688892 2499 5 29737 1 20030140 9718629 49 1 4 0 9688892 2500


Now, the last one is my workaround for the higher size when adding a column that do not have a lot of distinct values: just add a column before with more distinct values. Here I use the UNIQU one, but you probably have one that can be useful for your queries.

SQL> analyze index DEMO_RANGE_COVERING_WITH_PK validate structure offline;
Index DEMO_RANGE_COVERING_WITH_PK analyzed.
SQL> select height,blocks,lf_blks,lf_rows_len,br_rows,br_blks,br_rows_len,most_repeated_key,btree_space,used_space,pct_used,rows_per_key,blks_gets_per_access,opt_cmpr_pctsave,lf_uncmp_rows_len,lf_uncmp_blks from index_stats
 
HEIGHT BLOCKS LF_BLKS LF_ROWS_LEN BR_ROWS BR_BLKS BR_ROWS_LEN MOST_REPEATED_KEY BTREE_SPACE USED_SPACE PCT_USED ROWS_PER_KEY BLKS_GETS_PER_ACCESS OPT_CMPR_PCTSAVE LF_UNCMP_ROWS_LEN LF_UNCMP_BLKS
------ ------ ------- ----------- ------- ------- ----------- ----------------- ----------- ---------- -------- ------------ -------------------- ---------------- ----------------- -------------
3 2688 2565 9963892 2564 6 37456 1 20557908 10001348 49 1 4 2 9963892 2565

Now you get the idea. When creating an index, or adding columns for covering index, and you have the choice of column order, then try to have their first bytes selective enough so that the branch needs only a small substring to identify each leaf block (or lower level branches).

Block dumps

If you want to see the details about the branch length, here are some info from block dumps. I got them with the following:

SQL> column value new_value tracefile
SQL> select value from v$diag_info where name='Default Trace File';
VALUE
/u01/app/oracle/diag/rdbms/cdb1/CDB1/trace/CDB1_ora_6799.trc
SQL> exec for i in (select header_file, header_block from dba_segments where owner='DEMO' and segment_name='DEMO_RANGE') loop execute immediate 'alter system dump datafile '||i.header_file||' block '||(i.header_block+1); end loop;
PL/SQL procedure successfully completed.
SQL> host tail -20 &tracefile


Here is the last branch entry for the root block of DEMO_RANGE where the first column is not very selective and then the rowid is required in the branch:

row#351[3279] dba: 113261807=0x6c03cef
col 0; len 2; (2): c1 04
col 1; len 6; (6): 07 00 05 7b 00 25


Here is the last branch entry for the root block of DEMO_RANGE_COVERING_RANDOM where instead of the rowid the 3 first bytes of the RANDOM_TEXT column are sufficient:

row#3[8006] dba: 113263037=0x6c041bd
col 0; len 2; (2): c1 04
col 1; len 3; (3): 53 51 52
col 2; TERM


Here is the last branch entry for the root block of DEMO_RANGE_COVERING_CONSTANT where the full 80 bytes of CONSTANT_TEXT are not even sufficient, and the ROWID is needed as a 3rd column:

row#28[5316] dba: 117444566=0x7000fd6
col 0; len 2; (2): c1 04
col 1; len 80; (80):
78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78
78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78
78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78
78 78 78 78 78
col 2; len 6; (6): 07 00 05 43 00 25


Here is the last branch entry for the root block of DEMO_UNIQU_COVERING_CONSTANT where the first column is sufficient:

row#2[8026] dba: 117447160=0x70019f8
col 0; len 4; (4): c3 09 0d 04
col 1; TERM


So what?

We probably don’t need a feature like SQL Server INCLUDING indexes in most of the cases. However, this may require thinking about the order of columns, mainly:

• ensure that selective columns appear as early as possible (without compromising the index access efficiency of course) in order to lower the bytes required to address branches and leaves
• when adding columns, try to add first a column that will keep the clustering factor you had with the rowid, such as a date of insert

Added 14-APR-2018

The conclusion above was only focused at columns added for covering indexes (I wrote it after reading wrong things in this stackoverflow thread), and it is not a general statement about putting selective columns first, which is a common misconception. Columns like this CONSTANT_TEXT (which is an extreme case of non-selective) can have a better index compression (Enterprise Edition feature) when in front. Read this tread, answers and links from Richard Foote: https://twitter.com/OracleSK/status/984906294879539200

Cet article Covering indexes in Oracle, and branch size est apparu en premier sur Blog dbi services.

The IOUG Collaborate 18 is now done.
I presented 2 sessions there:

From Transportable Tablespaces to Pluggable Databases

The introduction comes from a 5 minutes talk at Oracle Open World 2016 in the ‘EOUC Database ACES Share Their Favorite Database Things’, on the history of having tablespaces self-contained (with a relative file number in 8.0 and locally managed tablespaces in 8.1). I’ve added a demo on a feature that is not well known – using RMAN to transport tablespaces without the need to have the source in read-only, available since 10g. And I demoed all PDB movement features in 12cR2, 12cR2 and 18c: remote clone, refreshable clones, PDB switchover, Online relocation,…

A full article on the topic is available on Oracle Scene: http://viewer.zmags.com/publication/07098028#/07098028/8 and feel free to gove feedback here if you are using those features. Lot of interesting comments went after the session.

12.2 Multitenant New Security Features to Clarify DevOps and DBA role separation

This session is basically a demo of lockdown profiles and resource manager settings at PDB level. With an introduction on DevOps because the goal of those features is to be able to lower the roundtrips between Dev and Ops by giving nearly full privileges on the PDB. Those features were developed by Oracle for their own managed cloud services: Exadata Express Cloud Service and Autonomous Data Warehouse. You are the administrator of your PDB there, but locked down to what cannot break the CDB, and limited to the resources you pay for.

I’ll give this session next Month in Dusseldorf at DOAG Datenbank: https://programm.doag.org/datenbank/2018/#/scheduledEvent/558645, so you still have the occasion to see how this Autonomous Data Warehouse Cloud service works from command line.

This is clearly an alternative to having Oracle Database on Docker, where containers have a clear separation between the user data and metadata (in the PDB) and the software distribution and data (in ORACLE_HOME, and in CDB$ROOT). But experience shows a slow adoption of multitenant, and developers are asking for Docker containers. But the separation is not so easy: it is clear that the user data must be in an external volume and the software (the Oracle Home – or at least the minimal part of it required to run the database and options). But a big part of the software (for example the dbms_… packages) is also in the database, in CDB$ROOT. Here again feel free to comment.

Cet article IOUG Collaborate 18 est apparu en premier sur Blog dbi services.

You want to try the Autonomous Database Cloud Service? That’s easy. Here is a Step-by-Step.

Cloud Credits

First, you need Cloud Credits. You may have bought them (any recent negotiation with Oracle Sales, even for on-premises, involves some Cloud Credits). You can have a free trial with 300$ Cloud Credits available for 1 month. To get another month, you need a different e-mail address and different Credit Card number (not charged). It is quite easy to have different e-mail addresses and your bank may provide virtual credit card where the number changes each time. Or you may have the 5000$ Cloud Credits available for 1 year from the Education program. I got those thanks to ACE Director program.

Update 01-MAY-2018 – There’s also the 500$ credits from the “white glove” program – you can ask to your Sales representative

In all cases you will be able to test the service without spending too much credits because:

• This service is not expensive ($2.5 per OCPU per Hour in Pay As You Go)
• It is very easy to start and stop the service, and then pay only for the hours where you connect
• If you choose ‘Bring You Own License’ in the creation, the per OCPU per Hour is only $0.48 (but be sure that you have covered
  See https://cloud.oracle.com/en_US/datawarehouse/pricing)
• And finally, during the trial promotion, the credits are consumed at discounted rate
  (after 9 hours of usage, I got less than 1$ used)

OCI Account

The first generation of Oracle Cloud, is now called ‘OCI Classic’, and you distinguish it when connecting as the Sign-In page mentions ‘Traditional Cloud Account’. You cannot access to ADWC with this account.

You need an access to the OCI (Oracle Cloud Infrastructure – the version 2 of Oracle Cloud).
If, when Sign-In, you are welcomed by this guy looking at his phone, you are at the right place. I’m always curious about how they choose an image for a page used every day and for several years. The oracle.com login page is easy with the headquarters blurry shot. For the OCI account, they choose the “Man On Smart Phone – Young Business Man Texting In Airport – Casual Urban Professional Businessman Using Smartphone App Smiling Happy Inside Office Building Or Airport” from the Adobe image stock.

Ashburn

For the moment, the ADWC service is available only in the Ashburn Cloud Center. Not yet in Europe (but planned for Frankfurt). You can see the regions here: https://cloud.oracle.com/data-regions. Then, when you receive your access to the Oracle Cloud Services, chose the Ashburn Data Center.

Update 01-MAY-2018 – It seems that the service is available in Frankfurt.

Create Instance

The instance creation is easy and fast. It will create a Pluggable Database (PDB) in the Oracle Cloud CDB. You provide a name, and ADMIN password (be careful, rule is at least 12 characters) which is the password you’ll use to connect as the ADMIN user. You can change it later and add new users. The Shape is different from the DBaaS here. You define the number of threads you want to use (it actually sets the CPU_COUNT for the PDB) and the size of PDB datafiles. You can change both later with Scale Up/Down.

PaaS Service Manager Command Line Interface

You can also create an ADWC service from the command line. I’ll show how to install and use PSM, the PaaS Service Manager Command Line Interface). Rodrigo Jorge has a nice description for DBaaS on his blog.

So, you download PSM:

curl -X GET -u my.cloud.account@me.com:MyP@ssw0rd -H X-ID-TENANT-NAME:idcs-31bbd63c3cb9466cb8a96f627b6b6116 https://psm.us.oraclecloud.com/paas/core/api/v1.1/cli/idcs-31bbd63c3cb9466cb8a96f627b6b6116/client -o psmcli.zip
 
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 86945 0 86945 0 0 16806 0 --:--:-- 0:00:05 --:--:-- 23820


The user:password are those you use in the account Sign-In.

The ‘Tenant Name’, you get it from the URL of this Man On Smart Phone Sign-in web page. You will see it also mentioned later as ‘Identity domain’ (like in OCI-Classic). If you have a doubt, create the service from the web console, click on it and you will see the Tenant Name.

So, you have a zip file and do not unzip it. It is a Python 3 module and you install it with ‘pip3′. You can do that in any OS.

I have the strange idea to run my laptop on Windows with Cygwin for command line stuff. Here are the python3 packages I have here.

Here is the installation of PDM:

pip3 install -U psmcli.zip
 
Processing ./psmcli.zip
...


And now the nice thing is that you will configure once your credentials with ‘psm setup’. You provide the user, password and tenant name (which is called ‘identity domain’ here):

$ psm setup
 
Username: my.cloud.account@me.com
Password: MyP@ssw0rd
Retype Password: MyP@ssw0rd
Identity domain: idcs-31bbd63c3cb9466cb8a96f627b6b6116
Region [us]:
Output format [short]:
Use OAuth? [n]:
----------------------------------------------------
'psm setup' was successful. Available services are:
 
o ADWC : Oracle Autonomous Data Warehouse Cloud
o ADWCP : Oracle Autonomous Data Warehouse Cloud Platform
o ANALYTICS : Oracle Analytics Cloud
o APICS : Oracle API Platform Cloud Service
o APICatalog : Oracle API Catalog Service
...


ADWC is on the list. You are ready to manage ADWC instances, such as create one:


$ psm adwc create-service -c - <<<' {
"serviceName": "ADWCx",
"adminPassword": "Ach1z00dAch1",
"numCpus": "1",
"storageCapacity": "1",
"serviceLevel": "PAAS",
"serviceVersion": "18.1.1",
"managedSystemType": "oracle",
"enableNotification": true,
"notificationEmail": "notifocations@me.com",
"isBYOL": true
} '
 
Message: Submitted job to create service [ADWCx] in domain [idcs-31bbd63c3cb9466cb8a96f627b6b6116].
Job ID: 25509908


We can check the status of job
$ psm adwc activities --service-name ADWC
Operation Type Status Start Time End Time
CREATE_SERVICE RUNNING 2018-04-28T19:57:31.056+0000 N/A

And a few minutes later the service is there:
$ psm adwc activities --service-name ADWC
Operation Type Status Start Time End Time
CREATE_SERVICE SUCCEED 2018-04-28T19:57:31.056+0000 2018-04-28T19:59:51.900+0000

We will see how to connect in a future post. Very easy from SQL Developer or SQLcl.

You can delete the service when you don’t need it anymore:

psm adwc delete-service --service-name ADWC


To save credits, you want an easy way to stop and start the service. That’s for tne next post as PSN requires a little hack there.

Cet article ADWC: Creation of Autonomous Database Cloud service est apparu en premier sur Blog dbi services.

In the previous post, I explained how to create an Autonomous Data Warehouse with PSM (PaaS Service Manager Command Line Interface). The most common operation you want to do with it is starting and stopping the service. This is the best way to save credits for hourly billed services. And PSM is the easiest: run from everywhere (it is Python 3) and no need to provide credentials each time. In the previous post, I explained how to setup PSM for the ADWC service.

Unfortunately, for starting and stopping the instance you may realize that:

• It is not in the documentation
• Syntax exists but doesn’t work

The documentation is there, but no mention of start-service, stop-service nor restart-service: https://docs.oracle.com/en/cloud/paas/autonomous-data-warehouse-cloud/user/adwc-commands.html

The online help has start/stop/restart-service:

$ psm adwc h
 
DESCRIPTION
Oracle Autonomous Data Warehouse Cloud
 
SYNOPSIS
psm ADWC [parameters]  
AVAILABLE COMMANDS
o services
List all Autonomous Data Warehouse Cloud instances
o service
List Autonomous Data Warehouse Cloud instance
o create-service
Provision Autonomous Data Warehouse
o delete-service
Unprovision Autonomous Data Warehouse
o scale-service
Scale Autonomous Data Warehouse
o start-service
This operation will set the operational state of service as started
o stop-service
This operation will set the operational state of service as stopped
o restart-service
This operation will set the operational state of service as after...
o view-backups
List all backups of Autonomous Data Warehouse Cloud instance
o view-backup
List a backup of Autonomous Data Warehouse Cloud instance
o backup
Backup Autonomous Data Warehouse
o view-restores
List all restore operations for Autonomous Data Warehouse Cloud instance
o view-restore
List a specified restore operation for Autonomous Data Warehouse Cloud...
o restore
Restore Autonomous Data Warehouse
o check-health
Health Check operation
o operation-status
View status of Autonomous Data Warehouse Cloud instance operation
o activities
View activities for Autonomous Data Warehouse Cloud instance
o help
Show help


All 3 take the same parameters, the service name, the REST API output format, and a boolean for wait of the completion of the job:

$ psm adwc start-service h
 
DESCRIPTION
This operation will set the operational state of service as started
 
SYNOPSIS
psm ADWC start-service [parameters] -s, --service-name
[-of, --output-format ] [-wc, --wait-until-complete ]  
AVAILABLE PARAMETERS
-s, --service-name (string)
Name of the Autonomous Data Warehouse Cloud instance
 
-of, --output-format (string)
Desired output format. Valid values are [short, json, html]  
-wc, --wait-until-complete (boolean)
Wait until the command is complete. Valid values are [true, false]. Default is
'false'.
 
EXAMPLES
psm ADWC start-service -s ExampleInstance


But…

So, the online help show it and I try it:

$ psm adwc start-service --service-name ADWC --output-format short -wc true
 
Error: Not Found. <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Draft//EN">
<HTML>
<HEAD>
<TITLE>Error 404--Not Found</TITLE>
</HEAD>
<BODY bgcolor="white">
<FONT FACE=Helvetica><BR CLEAR=all>
<TABLE border=0 cellspacing=5><TR><TD><BR CLEAR=all>
<FONT FACE="Helvetica" COLOR="black" SIZE="3"><H2>Error 404--Not Found</H2>
</FONT></TD></TR>
</TABLE>
<TABLE border=0 width=100% cellpadding=10><TR><TD VALIGN=top WIDTH=100% BGCOLOR=white><FONT FACE="Courier New"><FONT FACE="Helvetica" SIZE="3"><H3>From RFC 2068 <i>Hypertext Transfer Protocol -- HTTP/1.1</i>:</H3>
</FONT><FONT FACE="Helvetica" SIZE="3"><H4>10.4.5 404 Not Found</H4>
</FONT><P><FONT FACE="Courier New">The server has not found anything matching the Request-URI. No indication is given of whether the condition is temporary or permanent.</p><p>If the server does not wish to make this information available to the client, the status code 403 (Forbidden) can be used instead. The 410 (Gone) status code SHOULD be used if the server knows, through some internally configurable mechanism, that an old resource is permanently unavailable and has no forwarding address.</FONT></P>
</FONT></TD></TR>
</TABLE>
 
</BODY>
</HTML>


Unfortunately, this doesn’t work. Is it that those commands are not supported yet, reason why we don’t find them in the documentation? Or maybe the opposite: they do not work and rather than fix them, they removed them from the documentation. One thing I’m 100% sure: start-service and stop-service are the most useful commands for a CLI giving easy access to an hourly billed and I want them to work. And it is Python, JSON and HTML – nothing hidden there.

Hack Fix

The error message is about no matching URL. PSM metadata is stored in your user directory (~/.psm/data on Linux) with one JSON file for each Oracle platform service. Having a look at the URLs in ADWC.json the bug is obvious:

$ jq . ~/.psm/data/ADWC.json | grep uri
"uri": "/paas/api/v1.1/instancemgmt/{identityDomainId}/services/{serviceType}/instances",
"uri": "/paas/api/v1.1/instancemgmt/{identityDomainId}/services/{serviceType}/instances/{serviceName}",
"uri": "/paas/api/v1.1/instancemgmt/{identityDomainId}/services/{serviceType}/instances",
"uri": "/paas/api/v1.1/instancemgmt/{identityDomainId}/services/{serviceType}/instances/{serviceName}",
"uri": "/paas/api/v1.1/instancemgmt/{identityDomainId}/services/{serviceType}/instances/{serviceName}/scale",
"uri": "/paas/core/api/v1.1/instancemgmt/{identityDomainId}/services/{serviceType}/instances/{serviceName}/start",
"uri": "/paas/core/api/v1.1/instancemgmt/{identityDomainId}/services/{serviceType}/instances/{serviceName}/stop",
"uri": "/paas/core/api/v1.1/instancemgmt/{identityDomainId}/services/{serviceType}/instances/{serviceName}/restart",
"uri": "/paas/api/v1.1/instancemgmt/{identityDomainId}/services/{serviceType}/instances/{serviceName}/backups",
"uri": "/paas/api/v1.1/instancemgmt/{identityDomainId}/services/{serviceType}/instances/{serviceName}/backups/{backupId}",
"uri": "/paas/api/v1.1/instancemgmt/{identityDomainId}/services/{serviceType}/instances/{serviceName}/backups",
"uri": "/paas/api/v1.1/instancemgmt/{identityDomainId}/services/{serviceType}/instances/{serviceName}/restoredbackups",
"uri": "/paas/api/v1.1/instancemgmt/{identityDomainId}/services/{serviceType}/instances/{serviceName}/restoredbackups/{jobId}",
"uri": "/paas/api/v1.1/instancemgmt/{identityDomainId}/services/{serviceType}/instances/{serviceName}/restoredbackups",
"uri": "/paas/api/v1.1/instancemgmt/{identityDomainId}/services/{serviceType}/instances/{serviceName}/healthcheck",
"uri": "/paas/api/v1.1/activitylog/{identityDomainId}/job/{jobId}",
"uri": "/paas/api/v1.1/activitylog/{identityDomainId}/filter",


Let’s remove this ‘/core’ from the uri:

sed -ie 's/core\\[/]//g' ~/.psm/data/ADWC.json


And run again the start-service:

$ psm adwc start-service -s ADWC -wc true
 
Message: Job submitted successfully for start of service/system
Job ID: 25617877
Waiting for the job to complete... (it cannot be cancelled)


Here it is.

As my laptop is on Windows where I use Cygwin, I have setup two icons with:
C:\cygwin64\bin\mintty.exe -w min /usr/bin/psm adwc stop-service -s ADWC -wc true
C:\cygwin64\bin\mintty.exe -w min /usr/bin/psm adwc start-service -s ADWC -wc true

Start/Stop time

This ADWC service is a PDBaaS. Starting and Stopping is as easy as opening and closing a pluggable database. Here are the timestamps after starting and stoppin in a loop (with graph on 24 loops).

The startup time is around 30 seconds. The stop time is about a minute. Really easy to use.


$ psm adwc activities -s ADWC -l 50
Operation Type Status Start Time End Time
START_SERVICE SUCCEED 2018-04-29T14:30:36.888+0000 2018-04-29T14:31:21.563+0000
STOP_SERVICE SUCCEED 2018-04-29T14:27:26.551+0000 2018-04-29T14:27:35.610+0000
STOP_SERVICE SUCCEED 2018-04-29T14:25:22.172+0000 2018-04-29T14:25:51.586+0000
START_SERVICE SUCCEED 2018-04-29T14:20:47.957+0000 2018-04-29T14:21:38.131+0000
STOP_SERVICE SUCCEED 2018-04-29T14:08:09.409+0000 2018-04-29T14:08:48.125+0000
START_SERVICE SUCCEED 2018-04-29T14:07:24.892+0000 2018-04-29T14:08:08.244+0000
STOP_SERVICE SUCCEED 2018-04-29T14:04:57.566+0000 2018-04-29T14:05:27.458+0000
START_SERVICE SUCCEED 2018-04-29T14:03:51.035+0000 2018-04-29T14:04:34.108+0000
STOP_SERVICE SUCCEED 2018-04-29T14:03:17.701+0000 2018-04-29T14:03:47.262+0000
START_SERVICE SUCCEED 2018-04-29T14:02:00.944+0000 2018-04-29T14:02:50.978+0000
STOP_SERVICE SUCCEED 2018-04-29T14:00:56.990+0000 2018-04-29T14:01:29.567+0000
START_SERVICE SUCCEED 2018-04-29T13:59:52.898+0000 2018-04-29T14:00:39.373+0000
STOP_SERVICE SUCCEED 2018-04-29T13:59:19.380+0000 2018-04-29T13:59:49.011+0000
START_SERVICE SUCCEED 2018-04-29T13:58:15.594+0000 2018-04-29T13:58:58.937+0000
STOP_SERVICE SUCCEED 2018-04-29T13:57:42.355+0000 2018-04-29T13:58:11.845+0000
...


Easy command line without having to provide a password interactively, wait for completion, fast operation, this gives a great user experience for this service. The only problem is when you play with several cloud accounts. I’ll show an idea in the next post.

Cet article ADWC: start/stop with PSM Command Line Interface est apparu en premier sur Blog dbi services.