Thursday, March 29, 2012

Configuring OpenLDAP for High Availability. (Master/Slave or Provider/Consumer configuration) [Part 3 of 4]

This tech-note is the third in a series of 4, it describes the process to go through to setup openLDAP in a consumer/provider (master-slave) configuration. It is done in two steps here, the first step is setting up the two directories to replicate to each other which may be all that is required in some environments. The next step secures the consumer server so that it is accessed via LDAPS and then changes the replication configuration so that the replication is done over the encrypted channel.

Configuration of Consumer/Provider directory topology

Firstly we create an OpenLDAP directory instance on a compute node. Perform the steps described here to create a directory but stop short of running the migrate_all.pl script. (We do not need to create the entries in the directory as these will already exist in the master or Provider directory.)

i.e.

Edit the slapd.conf file to enter password details, suffix, directory location etc.
Create the directory location on the filesystem and give it ownership of ldap:ldap
Run updatedb, and locate DB_CONFIG and copy the config file into the directory server location.
On the provider create a user to use to for the replication traffic. This gives an LDAP user that can be used by the consumer to authenticate to the provider directory. I created an additional organisation unit (ou=service-users) to hold the users that are not unix OS users. Thus I have an entry as shown in the LDIF below:-

dn: cn=replication,ou=service-users,dc=oscexa,dc=com
objectclass: person
objectclass: top
cn: replication
sn: Replication User
userpassword: {SHA}41vs5sXm4OhspR0EQOkigqnWrIo=
Edit the slapd.conf file to add in all the details that will enable replication.

First change the slapd.conf on the provider putting the following lines at the end of the file (and the serverID up at the point in which the base info is defined.)

...

#######################################################################
# ldbm and/or bdb database definitions
#######################################################################

serverID 001database bdb
suffix "dc=el01,dc=com"
...

...

overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 100

Where

overlay syncprov - Specifies that the overlay syncprov is to be used. (Essentially an overlay is an add-on or plugin to extend openLDAP functionality. In the recent releases this overlay is compiled into the base openldap service so there is no need to specifically load the module.)
syncprov-checkpoint - Defines the number of operations or the number of minutes elapsed since the last checkpoint before checkpointing again. (eg. In our example the system will allow either 100 updates before checkpointing the system or 10 minutes.)
syncprov-sessionlog - The number represents the maximum number of session log entries.

Now change the slapd.conf on the consumer, again adding in the lines highlighed in bold.

...

index uidNumber,gidNumber,loginShell eq,pres
index uid,memberUid eq,pres,sub
index nisMapName,nisMapEntry eq,pres,sub

# Replicas of this database
#replogfile /var/lib/ldap/openldap-master-replog
#replica host=ldap-1.example.com:389 starttls=critical
# bindmethod=sasl saslmech=GSSAPI
# authcId=host/ldap-master.example.com@EXAMPLE.COM

syncrepl rid=001
provider=ldap://<The IP/hostname of provider server>:389
type=refreshAndPersist
searchbase="dc=oscexa,dc=com"
filter="(objectClass=*)"
scope=sub
attrs="*"
bindmethod=simple
binddn="cn=replication,ou=service-users,dc=el01,dc=com"
credentials=welcome1
tls_cert=/etc/openldap/cacerts/server.pem

mirrormode on
updateref ldap://<The IP/hostname of provider server>:389

The changes mentioned above are:-

ServerID - Adding a unique ID for each consumer server to add to the environment.
syncrepl rid=001 - This parameter and all the sub-parameters define the URL of the provider server to replicate from, what data is to be copied, the credentials of the user to use, how they bind, if LDAPS is to be used the location of the certificate file to use etc.
mirrormode - Defines that the server is to be a mirror of the provider and take over as the master should the original provider fail.
updateref - As this is a consumer any attempted updates are to be directed back to the provider.

Now restart the directory on the provider and then start the directory on the consumer . Check that the contents have been replicated over to the consumer by performing some queries against it.

Note - I ran into some oddities when setting this up myself in that I had added some additional security to my provider which limited the visibility of entries in the directory. Ensure that the user you are using to authenticate to the directory with has full visibility of the subtree required for authenticating Unix users. (In particular the Group and People subtrees.)

Configure Consumer for LDAPS

Setting up a consumer to use LDAPS is the same as described here. In summary:-

Create a self signed certificate and private key via the usage of the openssl command line
- # cd /etc/openldap/cacerts
- # openssl req -newkey rsa:1024 -x509 -nodes -out cacerts.pem -keyout slave-key.pem -days 3650
Edit the slapd.conf file to specify that the server is to start up an SSL listener. Using the cacerts.pem file as the CA certificate file and the slave.pem for encrypting & key.
- TLSCACertificateFile /etc/openldap/cacerts/cacerts.pem
- TLSCertificateFile /etc/openldap/cacerts/slave-key.pem
- TLSCertificateKeyFile /etc/openldap/cacerts/slave-key.pem
Edit the /etc/sysconfig/ldap file to specify that the server is to startup the LDAPS listener.
Restart the directory.
- service ldap restart

Test connecting to the directory to ensure that it has started up using the LDAPS self-signed certificate.

Configure Replication to use LDAPS

In this case the consumer will be connecting to the provider server using SSL encryption for the LDAP traffic. This means that the configuration for the replication must be changed to make use of the LDAPS URL and because we are using self-signed certificates we will have to include the public key from the provider server into the cacerts file that the consumer uses so that it will trust the certificate. Thus there are two steps to setting up secure replication:-

Copy the contents of the cacerts file on the provider machine (Everything from -----BEGIN CERTIFICATE....... to END CERTIFICATE -----) and add it to the consumer's TLSCACertificateFile. (Namely the /etc/openldap/cacerts/cacerts.pem file above.) Thus the Cert Authority (CA) Certificate's file contains the public keys for both the provider and the consumer directories.
Change the slapd.conf file to use the LDAPS protocol for connecting.

Thus the /etc/openldap/cacerts/cacerts.pem file on the consumer will look similar to:-

-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDjTCCAvagAwIBAgIJAK0aMmYpr7uXMA0GCSqGSIb3DQEBBQUAMIGMMQswCQYD
VQQGEwJHQjESMBAGA1UECBMJQmVya3NoaXJlMRAwDgYDVQQHEwdOZXdidXJ5MQ8w
DQYDVQQKEwZPcmFjbGUxDTALBgNVBAsTBEVTQVQxETAPBgNVBAMTCHZiZWxjbjAx
MSQwIgYJKoZIhvcNAQkBFhVkb24uZm9yYmVzQG9yYWNsZS5jb20wHhcNMTIwMzI5
MTU1MzMxWhcNMjIwMzI3MTU1MzMxWjCBjDELMAkGA1UEBhMCR0IxEjAQBgNVBAgT
CUJlcmtzaGlyZTEQMA4GA1UEBxMHTmV3YnVyeTEPMA0GA1UEChMGT3JhY2xlMQ0w
CwYDVQQLEwRFU0FUMREwDwYDVQQDEwh2YmVsY24wMTEkMCIGCSqGSIb3DQEJARYV
ZG9uLmZvcmJlc0BvcmFjbGUuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKB
gQCxKekSkIPiw7IDMzGYzC6aiDhC9rJIlJizziig4W+OgrpUqLpDaK2xmoSewD/V
RxCc8yjzPElI7YOcnb69M7rVfhPs8IEXl2YkW4qfy76FdiOwNcbRsoPk3WT9h69k
du8DgSddRvk537XhejCg0vpR/Lfj0U6tsuudVxIY+yWclwIDAQABo4H0MIHxMB0G
A1UdDgQWBBR8PO0bvXCJlARZbMHLu289yFzqCjCBwQYDVR0jBIG5MIG2gBR8PO0b
vXCJlARZbMHLu289yFzqCqGBkqSBjzCBjDELMAkGA1UEBhMCR0IxEjAQBgNVBAgT
CUJlcmtzaGlyZTEQMA4GAwDwYDVQQDEwh2YsY24wMTEkMCIGCSqGSIb3DQEJARYV
ZG9uLmZvcmJlc0BvcmFjbGUuY29tggkArRoyZimvu5cwDAYDVR0TBAUwAwEB/zAN
BgkqhkiG9w0BAQUFAAOBgQA+KrjdrkERBL4OaPib8BtEnLRMKCsgtKin0hbOJd+w
GJIr9BQNhYXB6qib2RWZBn9tF/7WfqLHavhaPgD3qo3d01TOWs2A09TaeX7FBk+g
Y4UU7QP9UarkZLSdEPfPuMmniCr8mrqRph/fH/qVVecU1U4mIVekQzanqd1vHii8
xg==
-----END CERTIFICATE-----
And the slapd.conf will have a section that looks similar to:-

...

# bindmethod=sasl saslmech=GSSAPI
# authcId=host/ldap-master.example.com@EXAMPLE.COM
syncrepl rid=001
provider=ldaps://<Hostname of provider LDAP server>:636
type=refreshAndPersist
searchbase="dc=el01,dc=com"
filter="(objectClass=*)"
...

In the same manner the update URL is switched to use the LDAPS URL as well.

Now edit the file /etc/sysconfig/ldap on both the provider and consumer servers and set them to only start up the LDAPS listener. The contents of the file looking like:-

# Parameters to ulimit called right before starting slapd
# - use this to change system limits for slapd
ULIMIT_SETTINGS=

# How long to wait between sending slapd TERM and KILL
# signals when stopping slapd by init script
# - format is the same as used when calling sleep
STOP_DELAY=3s

# By default only listening on ldap:/// is turned on.
# If you want to change listening options for slapd,
# set following three variables to yes or no
SLAPD_LDAP=no
SLAPD_LDAPS=yes
SLAPD_LDAPI=no

And now restart the directory services on both instances and test to ensure you can only access the services on the secure port (636) and that replication is working as you would expect.

# service ldap restart

Monday, February 20, 2012

Securing OpenLDAP Communications (LDAPS)

Securing OpenLDAP Communications (LDAPS)[part 2 of 4]

Introduction

Having setup a directory server and unix server clients it is now possible to authenticate OS users against the directory as a shared user repository for multiple nix based servers. However, we are not yet in a state that is ready for production, one issue to address is security. There are many aspects to security, this posting is only covering off one aspect, namely the transport level security where we change the configuration to make use of TLS to encrypt all data flowing in or out of the directory server so that it becomes impossible (well very hard) for someone to snoop the traffic on the network and discover a users' password.

In this example we make use of a self signed certificate to perform the encryption. This is fine for an internal directory server where we have control over all the clients and can set them up to trust the self-signed certificate. For production servers you should use a certificate from one of the trusted certificate authorities. The process for configuring openLDAP will be similar for self-signed or a certificate gained via a certificate authority. For details about self-signing or requesting from an authority see the openssl website and various blogs on the internet.

Creating a self signed certificate

We are using a self-signed certificate to perform the encryption. This can be done using the openssl tool, this is installed as part of the openssl package. In the distributions I have been using this has already been installed but if it is not present then use yum to install the package or download and add the rpm

# cd /etc/openldap/cacerts

# openssl req -newkey rsa:1024 -x509 -nodes -out server.pem -keyout server-key.pem -days 3650

answer the questions asked. An example shown below.

[root@c1718-10-100 cacerts]# openssl req -newkey rsa:1024 -x509 -nodes -out server.pem -keyout server-key.pem -days 3650
Generating a 1024 bit RSA private key
............................++++++
.................++++++
writing new private key to 'server.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:
State or Province Name (full name) [Berkshire]:
Locality Name (eg, city) [Newbury]:
Organization Name (eg, company) [My Company Ltd]:Your Company
Organizational Unit Name (eg, section) []:Your organisation
Common Name (eg, your name or your server's hostname) []:ldap-host
Email Address []:your.name@yourorg.com

This creates a self-signed certificate and key.

Configure OpenLDAP to use certificate for LDAPS communications.

Now we link this to OpenLDAP so that it uses the certificate to perform the encryption, enabling LDAPS. Simply add the following four lines to the slapd.conf file:-

...

TLSCACertificateFile /etc/openldap/cacerts/server.pem
TLSCertificateFile /etc/openldap/cacerts/server.pem
TLSCertificateKeyFile /etc/openldap/cacerts/server-key.pem
TLSCipherSuite HIGH:MEDIUM:+SSLv2

...

Now edit the file /etc/sysconfig/ldap and set up the machine so that it will start listening for LDAPS and stop listening for the plain text clients.

Contents of ldap file:-

# Parameters to ulimit called right before starting slapd
# - use this to change system limits for slapd
ULIMIT_SETTINGS=

# How long to wait between sending slapd TERM and KILL
# signals when stopping slapd by init script
# - format is the same as used when calling sleep
STOP_DELAY=3s

# By default only listening on ldap:/// is turned on.
# If you want to change listening options for slapd,
# set following three variables to yes or no
SLAPD_LDAP=no
SLAPD_LDAPS=yes
SLAPD_LDAPI=no

At this point is should now be possible to use any of the LDAP client browsers. (Apache Directory Studio, LDAP Browser, JXplorer etc.) to bind to the directory and query it's content via LDAPS. (The default port for LDAPS is 636, so the URL will be LDAPS://<your LDAP Host>:636.)

Client configuration to authenticate users via LDAP.

Now we need to configure the client side machines to authenticate via LDAP, using the secure LDAPS protocol. This is done via configuration in the files /etc/nsswitch.conf and /etc/ldap.conf. The former instructing the OS to use ldap to authenticate users and the latter providing the detail on how the server is to connect to LDAP.

For the nsswitch.conf file we simply include the check against ldap in the list of sources for passwd, group and shadow. Thus the file will look like:-

...
passwd: files ldap
shadow: files ldap
group: files ldap

...

Note - this file instructs the OS on the order of authentication methods, in this case the first attempt is to use the local file system (/etc/passwd etc.) and if the user is not registered there then lookup the user in LDAP. It is important to ensure that the order is - files then ldap. This ensures that should the directory be inaccessible it is still possible to authenticate and log onto the OS for management.

Now we need to make the necessary adjustments to the ldap.conf configuration file. The changes we do instruct the machine on where the directory is, what the base DN is, any authentication credentials required and the certificate trust store. The first step is to copy the server.pem file we created on the LDAP server over to this server. Then edit the pem file to remove the private key from the list of certificates. Copy the file into the directory /etc/openldap/cacerts

So the original file will look similar to :-

Remove the first "private key" entry to leave just the certificate:-

now edit the ldap.conf file to comment out the entries for the host:port and replace them with the entry for the uri, pointing at your directory server instance. We also have to specify that ssl is on and the location of the cacert file. Thus we end up with:-

#host 10.128.10.100

# The distinguished name of the search base.
base dc=oscexa,dc=com

# Another way to specify your LDAP server is to provide an
# uri with the server name. This allows to use
# Unix Domain Sockets to connect to a local LDAP Server.
#uri ldap://127.0.0.1/
uri ldaps://192.168.10.100:636
#uri ldapi://%2fvar%2frun%2fldapi_sock/
# Note: %2f encodes the '/' used as directory separator

...

# OpenLDAP SSL mechanism
# start_tls mechanism uses the normal LDAP port, LDAPS typically 636
#ssl start_tls
ssl on

# OpenLDAP SSL options
# Require and verify server certificate (yes/no)
# Default is to use libldap's default behavior, which can be configured in
# /etc/openldap/ldap.conf using the TLS_REQCERT setting. The default for
# OpenLDAP 2.0 and earlier is "no", for 2.1 and later is "yes".
#tls_checkpeer yes

# CA certificates for server certificate verification
# At least one of these are required if tls_checkpeer is "yes"
#tls_cacertfile /etc/ssl/ca.cert
#tls_cacertdir /etc/ssl/certs
tls_cacertfile /etc/openldap/cacerts/server.pem
...

Now check that the authentication is working as you would expect. i.e. You use a test user in the directory that is not on the local filesystem and authenticate to the OS as that user.

Time Synchronisation

When using SSL to secure communications it becomes important to ensure that your servers are all sync'd to the same time source. NTP is the simplest mechanism to use to perform this sync. To setup ntp simply edit the file /etc/ntp.conf and ensure that you put in the line to identify your NTP server. eg.

...

server <IP address of NTP server>

...

then restart your NTP daemon.

# service ntpd restart

You can check that things are working via the ntpq command line option.

# ntpq -n -c peers

     remote           refid      st t when poll reach   delay   offset  jitter

==============================================================================

 127.127.1.0     .LOCL.          10 l   53   64    1    0.000    0.000   0.001

 192.168.2.2      192.168.196.118  3 u   52   64    1    0.749  834.723   0.001

The numbers for the delay, offset and jitter should be non-zero. If they are showing up as 0 then the chances are that you are not able to access the NTP server correctly. If this is the case check the network access to the NTP server and that the server is responding to NTP requests. Secondly, check the current time on your server and compare it to the time on the NTP server. If the two are out by more than about 10 minutes then NTP assumes that things are very wrong and will not sync the time on the two machines.

Thursday, January 12, 2012

Using LDAP for Shared Authentication. (NFS v4 Requirement) [Part 1 of 4]

NFS v4 Requirements

NFS v3 is a stateless protocol so in order to achieve file locking it relies on the Network Lock Manager (NLM) to maintain consistency of any file. The NLM protocol in turn relies on the client IP address to keep a track of the owner of a particular lock. Because an Exalogic is a multi-homed machine (many IP addresses) it is possible for a lock request to come from one IP address and a release from another IP, this causes the NLM to not honour the request and a lock is not released. This causes problems with Weblogic as it is then unable to get the lock and hence fail to startup. Using NFS v4 resolves this problem because NFS v4 is a stateful protocol, the client being given a client ID during the initial negotiation phase and then re-using the same client ID for all subsequent requests. This means that the client IP address is not used to track lock ownership and so the impact of multiple IP addresses on the client is not an issue and hence Exalogic operates as desired using NFS v4.

NFS v4 is a fairly significant improvement to the NFS v3 protocol, in particular around the area of security where a client accessing the share also includes the user's security principle which is used to secure the file. This means that NFSv4 requires some form of shared repository of users so that it can check this to provide appropriate access to any particular file or directory. Two options are widly used in the industry for this functionality, historically the Network Information Service (NIS) also known as Yellow Pages (YP) the other option is to make use of a Directory Server via the Lightweight Directory Access Protocol or LDAP.

This posting explains how to install and get a very basic LDAP server operational and configure the Linux compute nodes and ZFS storage appliance of an Exalogic to use LDAP to access the users, passwords, hosts etc. from the directory.

Directory Server Architecture

In the first place a very simple non-HA approach is taken with one compute node of an Exalogic machine nominated as the host for the directory and all other nodes use its services to enable centralised authentication and management. Future postings will cover of setting up the directory to be multi-master/master-slave configurations so that the system becomes fault tolerant.

We will also be considering security, both at the transport layer where we will change the directory to use LDAPS so that it is not possibe to pick up transports from the network interchange and within the directory structure so that users have access to various combinations of compute nodes.

Prerequisites

First off create a group and user on your machine that we can use for testing the directory.

For example:-

# groupadd -g 500 donald

# useradd -g donald -d /home/donald -m -s /bin/bash -u 500 donald

Then login as this user to check that all is fine.

Installing openldap

The first step towards getting the shared repository setup is to get the directory server configured. In this example we are using openldap which is available as a package in the Oracle public Yum repository, alternatives include the likes of Oracle Internet Directory, Oracle Enterprise Directory Server, OpenDS etc. For openldap a number of packages must be installed onto the compute nodes. For the nodes that will host a directory server instance (Master or Slave) the following packages are required:-

openldap
openldap-clients
openldap-devel
nss_ldap
openldap-servers

and for the machines that will only operate as a client to the directory service the following packages should be installed:-

openldap
openldap-clients
openldap-devel
nss_ldap

In order to do this either setup your machine to connect to the Oracle public Yum repository or download the necessary packages.

For installing using yum:-

# yum install openldap-clients
Loaded plugins: security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package openldap-clients.x86_64 0:2.3.43-12.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
openldap-clients x86_64 2.3.43-12.el5 el5_u5_base 221 k

Transaction Summary
================================================================================
Install 1 Package(s)
Upgrade 0 Package(s)

Total download size: 221 k

Is this ok [y/N]: y
Downloading Packages:
openldap-clients-2.3.43-12.el5.x86_64.rpm | 221 kB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : openldap-clients 1/1

Installed:
openldap-clients.x86_64 0:2.3.43-12.el5

Complete!
# yum install openldap-devel

# yum install openldap-servers

(On my test systems openldap and nss_ldap packages had already been installed.)

Configuring the Directory Server

The main configuration files for openldap are all held in /etc/openldap. Open slapd.conf and set the value of the suffix and rootdn to something appropriate for the environment. This file also contains the directory on the file system that will be used to host the directory instance. By default this is set to /var/lib/ldap, change this to match your directory name.

You can also set the password for the Manager user by putting it into the slapd.conf file. This can be set unencrypted but should really be an encrypted value for security. Openldap includes the utility slappasswd which will encrypt a password entered and give the encrypted value back to standard out. Use this facility to encrypt your password and then put the value into the configuration file.

For example:-

[root@vbelcn01 openldap]# slappasswd
New password:
Re-enter new password:
{SSHA}Y0iT4bFZGz1QQ34NTjqwmc4N7BFpURsN

And copy the encrypted value into the slapd.conf file.

...

#######################################################################
# ldbm and/or bdb database definitions
#######################################################################

database bdb

suffix "dc=el01,dc=com"

rootdn "cn=Manager,dc=el01,dc=com"

rootpw {SSHA}Y0iT4bFZGz1QQ34NTjqwmc4N7BFpURsN
...

directory /var/lib/ldap/el01.com

...

Now create the directory that will be used to host the Directory Server instance and set the ownership to the ldap user. (The ldap user is created when the packages are installed.)

# mkdir /var/lib/ldap/el01.com

# chown ldap:ldap /var/lib/ldap/el01.com

Create the directory and populate with existing values

The directory is to be used to manage the users, groups, hosts etc., openldap comes with a number of scripts that can be used to copy existing values into the directory. These scripts are installed into /usr/share/openldap/migration. Start by editing the migrate_common.ph script and replace all instances of the domain padl.com with the domain that you are using for your directory.

For example, in our case we are using the domain el01.com so substitute all string occurances of "padl" with "el01".

Run updatedb then locate DB_CONFIG to create the configration file for the database.

[root@vbelcn01 el01.com]# updatedb
[root@vbelcn01 el01.com]# locate DB_CONFIG
/etc/openldap/DB_CONFIG.example
[root@vbelcn01 el01.com]# cp /etc/openldap/DB_CONFIG.example /var/lib/ldap/el01.com/DB_CONFIG

Run the migrate_all_offline.ph script to populate the directory with the values from your various files.

[root@vbelcn01 tmp]# cd /usr/share/openldap/migration

[root@vbelcn01 migration]# ./migrate_all_offline.sh
Creating naming context entries...
Migrating groups...
Migrating hosts...
Migrating networks...
Migrating users...
Migrating protocols...
Migrating rpcs...
Migrating services...
Migrating netgroups...
Importing into LDAP...
Migrating netgroups (by user)...
Migrating netgroups (by host)...
Preparing LDAP database...
=> bdb_tool_entry_put: id2entry_add failed: DB_KEYEXIST: Key/data pair already exists (-30996)
=> bdb_tool_entry_put: txn_aborted! DB_KEYEXIST: Key/data pair already exists (-30996)
slapadd: could not add entry dn="dc=vbel,dc=com" (line=5): txn_aborted! DB_KEYEXIST: Key/data pair already exists (-30996)
Migration failed: saving failed LDIF to /tmp/nis.ldif.z17552

The error reported is linked to the services available on the server and as we are not planning on using these via LDAP lookups the error does not seem to impact the behaviour of the directory. Be warned, if you are doing this in an environment that you have been using for a while take care to ensure you have no duplicate entries in your hosts file. If you do then the migrate script fails to update the directory due to a duplicate entry and the script will stop. The error reported will be similar to:-

bdb_tool_entry_put: id2entry_add failed: DB_KEYEXIST: Key/data pair already exists (-30996) 

bdb_tool_entry_put: txn_aborted! DB_KEYEXIST: Key/data pair already exists (-30996)

slapadd: could not add entry dn="cn=exatest2.oracle.com,ou=Hosts,dc=el01,dc=com" (line=469): txn_aborted! DB_KEYEXIST: Key/data pair already exists (-30996)

Migration failed: saving failed LDIF to /tmp/nis.ldif.DA3373

Now we want to start up the directory. Prior to issuing the command to start the daemon we must change all the files in the directory to be owned by ldap:ldap.

[root@vbelcn01 migration]# cd /var/lib/ldap/el01.com/
[root@vbelcn01 el01.com]# ls -l
total 3312
-rw-r--r-- 1 root root 2048 Jan 4 11:00 alock
-rw------- 1 root root 122880 Jan 4 10:59 cn.bdb
-rw------- 1 root root 24576 Jan 4 11:00 __db.001
-rw------- 1 root root 368640 Jan 4 11:00 __db.002
-rw------- 1 root root 270336 Jan 4 11:00 __db.003
-rw------- 1 root root 98304 Jan 4 11:00 __db.004
-rw------- 1 root root 557056 Jan 4 11:00 __db.005
-rw------- 1 root root 24576 Jan 4 11:00 __db.006
-rw-r----- 1 root root 921 Jan 4 10:54 DB_CONFIG
-rw------- 1 root root 86016 Jan 4 10:59 dn2id.bdb
-rw------- 1 root root 12288 Jan 4 10:59 gidNumber.bdb
-rw------- 1 root root 245760 Jan 4 10:59 id2entry.bdb
-rw------- 1 root root 10485760 Jan 4 11:00 log.0000000001
-rw------- 1 root root 8192 Jan 4 10:59 loginShell.bdb
-rw------- 1 root root 8192 Jan 4 10:59 memberUid.bdb
-rw------- 1 root root 8192 Jan 4 10:59 nisMapName.bdb
-rw------- 1 root root 36864 Jan 4 10:59 objectClass.bdb
-rw------- 1 root root 8192 Jan 4 10:59 ou.bdb
-rw------- 1 root root 20480 Jan 4 10:59 uid.bdb
-rw------- 1 root root 8192 Jan 4 10:59 uidNumber.bdb
[root@vbelcn01 el01.com]# chown -R ldap:ldap *

Now start the DB and ensure it starts on boot.

[root@vbelcn01 vbel.com]# service ldap start
Checking configuration files for slapd: config file testing succeeded
[ OK ]
Starting slapd: [ OK ]
[root@vbelcn01 vbel.com]# chkconfig ldap on
[root@vbelcn01 vbel.com]# chkconfig --list ldap
ldap 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Then check that it is accessible and contains the values expected by issuing an ldapsearch.

[root@vbelcn01 vbel.com]# ldapsearch -x -b "ou=People,dc=vbel,dc=com" '(uidNumber=500)'
# extended LDIF
#
# LDAPv3
# base <ou=People,dc=vbel,dc=com> with scope subtree
# filter: (uidNumber=500)
# requesting: ALL
#

# donald, People, vbel.com
dn: uid=donald,ou=People,dc=vbel,dc=com
uid: donald
cn: Donald Forbes
objectClass: account
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
userPassword:: e2NyeXB0fSQxJHNtMHNQRVpLJG9XUm5VdGkzNnJJc0NkOEVMNTU5Mi4=
shadowLastChange: 15342
shadowMin: 0
shadowMax: 99999
shadowWarning: 7
loginShell: /bin/bash
uidNumber: 500
gidNumber: 500
homeDirectory: /home/donald
gecos: Donald Forbes

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
[root@vbelcn01 vbel.com]#

Note - in this case I narrowed the search criteria down to specify my own user record that is identified with the UID of 500. The search criteria can be wider to return all the hosts, users, services etc.

[root@vbelcn01 vbel.com]# ldapsearch -x -b "dc=vbel,dc=com" '(objectclass=*)'

Configuring a client to use LDAP

First edit the /etc/ldap.conf file to specify the directory server host IP address and the base DN used.

...

host 192.168.23.105

base dc=el01,dc=com

...

Edit the /etc/nsswitch.conf file to specify all the entries that you would like to have resolved in ldap, specifying the resolution order. For example:-

...

passwd: files ldap
shadow: files ldap
group: files ldap
...

Setup the identity mapping ready for mapping NFS v4 users. This is managed via the idmapd daemon in Linux. Edit the /etc/idmapd.conf file and specify the domain name you wish to use. For example setting it to el01 would result in an idmapd.conf file that looks like:-

[root@vbelcn01 etc]# cat /etc/idmapd.conf
[General]

Verbosity = 0
Pipefs-Directory = /var/lib/nfs/rpc_pipefs
Domain = el01

[Mapping]

Nobody-User = nobody
Nobody-Group = nobody

[Translation]
Method = nsswitch

Now restart the idmapd service.

# service rpcidmapd stop

# service rpcidmapd start

Test your users

Firstly check that you can log in as the user you created at the beginning of this exercise. If all is OK then we can can continue to create a new user in LDAP and see if we can authenticate as this user.

There are various command line tools that can be used to add users, modify them etc. - ldapadd ldapsearch etc. However for a test system you may well have access to the GUI tools, there are many available but for flexibility I have found the Apache studio to be very useful. (http://directory.apache.org/studio/) Using one of these tools create a user similar to the one you created using the unix command useradd. (I exported the my user to an LDIF file, edited the file to change the dn, cn, uidNumber, uid and password then imported the changed ldif back into the directory. This approach means all the objectClass requirements are met.)

Having got the new entry in the directory then attempt to log in as that user, change its password and logout then re-authenticate. (Note - If you are having a problem authenticating try setting the password in the directory to use crypt with a new salt.)

Setup the Shared Storage to use LDAP Authentication

Select the LDAP service and specify the root dn to search from, namely dc=el01,dc=com assuming this was the suffix you are using in your directory server. Select the Search Scope to be "sub-tree" which is a recursive search and will hunt all units below the base dn. Now click on the + sign by the servers to put in the IP address and port of the server running the directory.

Enable the LDAP service.

To configure the ZFS storage appliance so that it uses LDAP simply ensure that the NIS service is turned off and the NFS service has the option to use the DNS domain unselected. Then you need to put into the custom identity domain field the value of the domain as used in the /etc/idmapd.conf file on each client compute node. The default name for this is localdomain but it is recommended that you switch this to reflect your environment - eg. el01.

Testing the Shared Storage is configured correctly.

Having made the configuration changes to the shared storage the simplest mechanism to ensure that it has been configured correctly is to create a project and a share, giving the share "Root Directory Access" of one of the users & group that you have setup in the directory server. The software complains if the username/group are invalid. If this is Ok you can double check by "shelling" out to the command line and list the directory you have just created for the share, the file permissions should match the user/group you have in LDAP.

For example, the configuration details for a test share on /export/ldaptest/test are shown below:-

vbzfs:> shares
vbzfs:shares> select ldaptest
vbzfs:shares ldaptest> select test
vbzfs:shares ldaptest/test> ls
Properties:
aclinherit = restricted (inherited)
atime = true (inherited)
casesensitivity = mixed
checksum = fletcher4 (inherited)
compression = off (inherited)
dedup = false (inherited)
compressratio = 100
copies = 1 (inherited)
creation = Wed Jan 11 2012 12:19:52 GMT+0000 (UTC)
logbias = latency (inherited)
mountpoint = /export/ldaptest/test (inherited)
normalization = none
quota = 0
quota_snap = true
readonly = false (inherited)
recordsize = 128K (inherited)
reservation = 0
reservation_snap = true
secondarycache = all (inherited)
shadow = none
nbmand = false (inherited)
sharesmb = off (inherited)
sharenfs = sec=sys,anon=ldaptest,rw=@192.168.23.0/24,root=@192.168.23.0/24 (inherited)
snapdir = hidden (inherited)
utf8only = true
vscan = false (inherited)
sharedav = off (inherited)
shareftp = off (inherited)
sharesftp = off (inherited)
sharetftp = (inherited)
pool = exalogic
canonical_name = exalogic/local/ldaptest/test
exported = true (inherited)
nodestroy = false
space_data = 31K
space_unused_res = 0
space_snapshots = 0
space_available = 69.9G
space_total = 31K
root_group = donald
root_permissions = 755
root_user = ldaptest
origin =

Children:
snapshots => Manage snapshots
replication => Manage remote replication
users => View per-user usage and manage user quotas
groups => View per-group usage and manage group
quotas
shadow => Manage shadow data migration

vbzfs:shares ldaptest/test>

Testing the NFS version 4 mounts

Now we are all ready to go and we would like to mount the nfs v4 shares. To do this persistently we need to firstly create the directory where we will mount the share.

# mdkir -p /u01/ldaptest/test

And then add an entry to the /etc/fstab file to specify the mount point:

vbzfs:/export/ldaptest/test /u01/ldaptest/test nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072 0 0

This can be mounted with the command

# mount -a

and unmounted

# umount -a -t nfs4 (to unmount all nfs4 partitions listed in the /etc/fstab file.)

# umount /u01/ldaptest/test (to unmount the one specific partition.)

Need to Remount File Systems After System Reboots When NFSv4 Is Used

When you use NFSv4, you are required to re-mount all file systems by running the mount -a command on Oracle Linux each time you reboot the compute node.

To automatically re-mount the file systems rebooting a compute node running Oracle Linux, complete the following steps:

1.Log in to the compute node as a root user.

2.Create a script named nfs4_automount_on_init in /etc/init.d/ directory (Log in as root user)

# vi /etc/init.d/ nfs4_automount_on_init

3.Add the following to the script:

#!/bin/bash#chkconfig: 345 80 05
#description: NFSv4 Automount
case "$1" in
"start")
mount -a
;;
"stop")
;;
esac
Note:

In chkconfig: 345 80 05, the first number 345 indicates the runLevels. The second number 80 is the start priority, and the third number 05 is the stop priority. The second number and the third number should be unique among all registered services, so you may need to adjust the values for your environment, if you have installed any other services.

4.Set up executable permission on the script:

# chmod +x /etc/init.d/ nfs4_automount_on_init

5.Run the following command to register the script:

# chkconfig --add nfs4_automount_on_init

6.Optionally, run the following command to verify the registration of the script:

# chkconfig --list

Tuesday, December 20, 2011

Welcome to another, technically focused, blog. I will be writing about some general IT and unix related topics but as my current job is to work on the Oracle Engineered Systems so most postings will relate to these machines and the software that runs on them. In particular Oracle Linux 5 or Solaris 11 running Weblogic and Coherence. Those familiar with the Exalogic will know that it is quite capable of running other software, in particular the full Oracle stack but essentially any software certified for OEL 5 or Solaris (64 bit for both) will run successfully on the platform.

There are a number of machines now available in the Oracle engineered system family. These being:-

Exadata
First machine out the door and aimed at running the Oracle database optimally.
Exalogic
The "middleware" machine. Optimised for Weblogic, Coherence and Tuxedo in the first place with the Oracle apps following on.
SPARC SuperCluster
Exalogic for the fans of the SPARC processor. Bringing the power of the T4 chip together with the ingenuity of the Exadata storage cells so that this machine covers all tiers in a typical enterprise deployment.
Exalytics
Perfect machine for running in-memory analysis of data.
Oracle Database Appliance
For those who don't need all the power of Exadata but do want the simplicity and manageability of a database appliance.
Oracle Big Data
Moving away from the structured data of the traditional SQL database into the world of gathering unstructured data and analysing this. Picking up feeds from the likes of social media applications.

First a very quick summary of what you get when you unwrap the covers from your shiny new engineered system. (For the two I am most interested in - Exalogic and SSC )

Exalogic (1 rack)
Up to 30 x86-64 compute nodes. (12 cores per node)
A ZFS storage appliance. 40Tb raw or ~18Tb actual once mirroring, spares etc. has been done.
Up to 4 Inifiniband switches to give a superfast backplane network across the machine (These are used to bridge out to a 10Gb Ethernet network for client access.)
1 Cicso 1GbE switch for management
Sparc Super Cluster (SSC)
Up to 4*T4-4 compute nodes
Same ZFS storage capability as the Exalogic machine
Infiniband networking
Up to 6 Exadata storage cells