Testing ASM on Oracle 11G with no external disks

Purpose:
Install and configure ASM (Automatic Storage Management) with Oracle 11G with no external disks.

Description:

Some good ASM features can be listed and we might need to do some testing before migrating to a Production environment. In order to simulate ASM with Oracle 11G with no external disks, I'll try ASM installation and configuration on a single node Linux system. Red Hat Enterprise Linux ES release 4 (Nahant Update 4) with kernel 2.6.9-42.ELsmp x86_64 x86_64 x86_64 GNU/Linux.

• ASM takes control of disk devices (no disk partitioning needed)
• More space can be transparently added
• I/Os are spread over all the available disks (striping)
• ASM can maintain redundant copies of the data (mirror)
• Storage can be migrated between disks
• Same disk (or set of disks) can be shared amongst many DBs
  

Oracle Version: Oracle Database 11g (11.1.0.6.0)

Process:

I'll simulate external disks with system files, using the loopback device. Loop devices see a regular file as a device. (I'll need system root privileges)

1. - ASM needs the Oracle cluster Synchronization Service (CSS)
run $ORACLE_HOME/bin/localconfig add (root user)

[root@tauro oracle] # $ORACLE_HOME/bin/localconfig add
/etc/oracle does not exist. Creating it now.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configuration for local CSS has been initialized

Cleaning up Network socket directories
Setting up Network socket directories
Adding to inittab
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
tauro
Cluster Synchronization Services is active on all the nodes.
Oracle CSS service is installed and running under init(1M)

2.- Install asmlib 2.0
Download and install the correct library version from http://www.oracle.com/technology/software/tech/linux/asmlib/rhel4.html (root)

[root@tauro ASM]# rpm -ivh oracleasm-2.6.9-42.ELsmp-2.0.3-1.x86_64.rpm
Preparing... ########################################### [100%]
1:oracleasm-2.6.9-42.ELsm########################################### [100%]

[root@tauro ASM]# rpm -ivh oracleasm-support-2.0.3-1.x86_64.rpm
Preparing... ########################################### [100%]
1:oracleasm-support ########################################### [100%]

[root@tauro ASM]# rpm -ivh oracleasmlib-2.0.2-1.x86_64.rpm
Preparing... ########################################### [100%]
1:oracleasmlib ########################################### [100%]

3.- Configure asmlib 2.0
(root)
[root@tauro ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: [ OK ]
Creating /dev/oracleasm mount point: [ OK ]
Loading module "oracleasm": [ OK ]
Mounting ASMlib driver filesystem: [ OK ]
Scanning system for ASM disks: [ OK ]

4.- Create virtual disks
(root)
(http://www.orafaq.com/node/1849)

dd if=/dev/zero of=disk1 bs=1024k count=2000
dd if=/dev/zero of=disk2 bs=1024k count=2000
dd if=/dev/zero of=disk3 bs=1024k count=2000
dd if=/dev/zero of=disk4 bs=1024k count=2000

5.- Using the loopback device to assign disks.
(root)
[root@tauro ASM]# /sbin/losetup /dev/loop1 /datos/oradata/catalogo/ASM/disk1
[root@tauro ASM]# /sbin/losetup /dev/loop2 /datos/oradata/catalogo/ASM/disk2
[root@tauro ASM]# /sbin/losetup /dev/loop3 /datos/oradata/catalogo/ASM/disk3
[root@tauro ASM]# /sbin/losetup /dev/loop4 /datos/oradata/catalogo/ASM/disk4

6.- Create ASM disks
(root)
[root@tauro ~]# /etc/init.d/oracleasm createdisk ASM1 /dev/loop1
Marking disk "/dev/loop1" as an ASM disk: [ OK ]
[root@tauro ~]# /etc/init.d/oracleasm createdisk ASM2 /dev/loop2
Marking disk "/dev/loop2" as an ASM disk: [ OK ]

[root@tauro dev]# /etc/init.d/oracleasm createdisk ASM3 /dev/loop3
Marking disk "/dev/loop3" as an ASM disk: [ OK ]
[root@tauro dev]# /etc/init.d/oracleasm createdisk ASM4 /dev/loop4
Marking disk "/dev/loop4" as an ASM disk: [ OK ]

/dev/oracleasm/disks
[oracle@tauro disks]$ ll
total 0
brw-rw---- 1 oracle dba 7, 1 May 21 12:10 ASM1
brw-rw---- 1 oracle dba 7, 2 May 21 12:10 ASM2
brw-rw---- 1 oracle dba 7, 3 May 21 12:11 ASM3
brw-rw---- 1 oracle dba 7, 4 May 21 12:11 ASM4

7.- Verify

(Use the DBCA utility to create the ASM instance on Oracle).

$ export ORACLE_SID=+ASM

$ sqlplus /nolog

SQL> conn / as sysdba
Connected.
SQL> set lines 132
SQL> col PATH for a50

SQL> select path, mount_status from v$asm_disk;

SQL> select group_number, disk_number ,name, state, mode_status, mount_status, total_mb from v$asm_disk;

SQL> select group_number GROUP#, name, state, total_mb, free_mb, required_mirror_free_mb REQ_MIRR_FREE_MB, usable_file_mb from v$asm_diskgroup;

8.- Add to startup
(root)

Add the following entries to the file "/etc/rc.local"

/sbin/losetup /dev/loop1 /datos/oradata/catalogo/ASM/disk1
/sbin/losetup /dev/loop2 /datos/oradata/catalogo/ASM/disk2
/sbin/losetup /dev/loop3 /datos/oradata/catalogo/ASM/disk3
/sbin/losetup /dev/loop4 /datos/oradata/catalogo/ASM/disk4
/etc/init.d/oracleasm createdisk ASM1 /dev/loop1
/etc/init.d/oracleasm createdisk ASM2 /dev/loop2
/etc/init.d/oracleasm createdisk ASM3 /dev/loop3
/etc/init.d/oracleasm createdisk ASM4 /dev/loop4

Note: See online reference on asmcmd command line tool

Links, References:

http://www.oracle.com/technology/products/database/asm/index.html
http://www.orafaq.com/wiki/Automatic_Storage_Management
http://www.orafaq.com/node/1849
http://en.wikipedia.org/wiki/Loop_device
http://www.oracle.com/technology/software/tech/linux/asmlib/rhel4.html
http://skrajend.blogspot.com/2007/06/simulating-asm-on-10g-by-faking_22.html

http://www.oracle.com/technology/oramag/oracle/06-mar/o26asm.html

Preparing OS for Installing Oracle 11gR1 64bit RDBMS on RHEL 4 UPD4

Regarding "Preparing OS for Installing Oracle 11gR1 32bit RDBMS on OEL 5 & RHEL 5" post, I've attempted the same process on 64Bit so as to take advantage of an existing Linux installation.

I've followed Metalink notes:
437123.1 (Requirements for Installing Oracle 11gR1 RDBMS on RHEL 4 on AMD64/EM64T) Revision 05-NOV-2007
169706.1 Revision 30-JAN-2008

I’ve started the installation on an IBM eServer xSeries x3650 with 2048MB and 2 Intel(R) Xeon(R) 5160 @ 3.00GHz, CPUs. The box has installed a default Red Hat Enterprise Linux ES release 4 (Nahant Update 4) (Linux 2.6.9-42.ELsmp #1 SMP x86_64 x86_64 x86_64 GNU/Linux)

This procedure is meant for those planning/trying to install Oracle Server 11.1.0.x.0 on Redhat 4 on the x86_64 (64-bit) platform. Since it is the expressed goal to keep Oracle Enterprise Linux functionally IDENTICAL to RHEL, this NOTE is also completely applicable to OEL 4 (update 4 or higher).

Info:
Automatic Memory Management.
Starting with Oracle Database 11g, the Automatic Memory Management feature requires more shared memory (/dev/shm) and file descriptors. The shared memory should be sized to be at least the greater of MEMORY_MAX_TARGET and MEMORY_TARGET for each Oracle instance on the computer. To determine the amount of shared memory available, enter the following command: # df -k /dev/shm/
In our case:

[root@xxxxx ~]# df -k /dev/shm/
S.ficheros Bloques de 1K Usado Dispon Uso% Montado en
none 1028096 0 1028096 0% /dev/shm

Note: MEMORY_MAX_TARGET and MEMORY_TARGET cannot be used when LOCK_SGA is enabled or with huge pages on Linux

Required RPMS
I've found a different list of required RPMs on both notes (437123.1, 169706.1)
Some RPMs where placed on system with the OS installation. Next graphic shows a detailed list and result:

TABLE 1 (click for zoom it)

Note: i386 packages might require the --force option during installation if the 64-bit version of the same package is already installed. For example, "rpm -ivh --force and could be found on both Red Hat updates (U4 & U5)

A list of required RPMs:
binutils-2.15.92.0.2
gcc-3.4.5
gcc-c++-3.4.5
gcc-3.4.6-8.i386.rpm
cpp-3.4.6-8.i386.rpm
libgcc-3.4.5 (32-bit)
libgcc-3.4.5
compat-libstdc++-33-3.2.3
compat-libstdc++-33-3.2.3 (32 bit)
elfutils-libelf-0.97
elfutils-libelf-devel-0.97
glibc-2.3.4-2.19
glibc-2.3.4-2.19 (32 bit)
glibc-common-2.3.4
glibc-devel-2.3.4
glibc-devel-2.3.4 (32-bit)
libaio-0.3.105
libaio-0.3.105 (32 bit)
libaio-devel-0.3.105
libstdc++-3.4.5
libstdc++-3.4.5 (32 bit)
libstdc++-devel 3.4.5
make-3.80
sysstat-5.0.5
glibc-headers-2.3.4-2.19
glibc-kernheaders-2.4-9.1.98.EL

*see Table 1

Note: to distinguish between a 32-bit or 64-bit package
i.e. #rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n" | grep libaio-0.3

Setting Kernel parameters and environment.

* Required Environment Variable

umask for the oracle account must be 0022
the "ulimit" stack size value should be at least 32768 (ulimit -s 32768)

* Modify your kernel settings in /etc/sysctl.conf (RedHat) as follows. If the current value for any parameter is higher than the value listed in this table, do not change the value of that parameter. Range values (such as net.ipv4.ip_local_port_range) must match exactly.

kernel.shmall = 2097152
kernel.shmmax = 1/2 of physical RAM. This would be the value 2147483648 for a 4Gb RAM system.
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 512 x processes (for example 65536 for 128 processes)
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 262144

Local values:
# Oracle
kernel.shmall = 2097152
kernel.shmmax = 1052770304
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 76800
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 262144

Set Shell Limits for the oracle User. Assuming that the "oracle" Unix user will perform the installation, do the following:

- Add the following settings to /etc/security/limits.conf

oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

- Add or edit the following line in the /etc/pam.d/login file, if it does not already exist:
session required pam_limits.so

- Add the following lines to /etc/profile:

if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi

Note: User settings are best described on "Preparing OS for Installing Oracle 11gR1 32bit RDBMS on OEL 5 & RHEL 5"

Installation note:

I’ve found an OS issue, launching runinstaller from linux.x64_11gR1_database.zip binaries.

./runInstaller

"You are attempting to install 64-bit Oracle on a 32-bit operating system. This is not supported and will not work."

It’s not true due the actual OS

[oracle@xxxxx bin]$ uname -mi

x86_64 x86_64

Problem, getconf shows 32 bit on a 64 bit system (maybe it's concerning Intel Extended Memory 64 Technology (Intel Intel64) that enables servers based on Intel Xeon processors to run both 32-bit and 64-bit applications under 64-bit operating systems):

[oracle@xxxxx bin]$ getconf LONG_BIT

32

Then, I’ve commented the following lines on “RunInstaller” and continue installation without problems:

#if [ `$UNAME` = "Linux" ]; then

# if [ -e $GETCONF ]; then

# value=`$GETCONF LONG_BIT`

# if [ $value != 64 ]; then

# echo "\"You are attempting to install 64-bit Oracle on a 32-bit operating system. This is not supported and will not work.\"";

# exit 0;

# fi

# fi

#fi

References:

Oracle Metallink Note: 437123.1
Oracle Metallink Note: 169706.1