Well, I learned another valuable clustering lesson today. Manually assign all of your IP addresses and do not rely on DHCP. I'd still suggest you have the IPs statically assigned on the DHCP server but don't rely on it at all.

Yesterday I rebooted all of the machines in the cluster simultaneously after making some extensive modifications to the iSCSI configuration on them all. (I'm using cluster-ssh to manage them all simultaneously.) All of the machines went down at exactly the same time and came up at exactly the same time. (They are identical hardware with identical software, afterall.)

Today, for whatever reason, when all of the leases expired at roughly the same time they didn't renew quickly enough. I walked out of a meeting this afternoon to all of the cluster nodes reporting "Quorum dissolved" and all services on the cluster were failed.

Labels: cluster, dhcp, linux, red hat enterprise linux