Red Hat CLUSTER SUITE FOR ENTERPRISE LINUX 4.5 Manuel d'utilisateur

Xin Chen, Garima Kochhar and Mario Gallegos Dell HPC Engineering Version 2.1, May 2012 Dell HPC NFS Storage Solution High Availability

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 10 Appendix A: NSS-HA Recipe includes detailed instructions on t

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 11 NFS server configuration in NSS-HA Figure 4. The NSS-HA arc

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 12 2) Step 2 – Logical volume configuration In this step, a log

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 13 3.2. Potential failures and fault tolerant mechanisms in NSS

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 14 New components and updates from previous versions of the 4.s

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 15 4.2. Storage configuration In previous versions of the solut

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 16 release, do not refer to the instructions listed in the previ

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 17 New server components in this release Table 4.Server compon

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 18 5.4 describes these functionality tests and their results. Fu

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 19 NSS-HA test bed Figure 7.

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities ii This document is for informational purpose

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 20 NSS-HA hardware configuration details Table 5.

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 21 NSS-HA software configuration details Table 6.SOFTWARE Oper

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 22 NSS-HA client configuration details Table 8.Client / HPC Co

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 23 NFS Server configuration 7) The XFS file system is mounted w

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 24 2) Heartbeat link failure - simulated by disconnecting the p

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 25 Impact to clients Clients mount the NFS file system exported

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 26 Metadata tests were performed using the mdtest benchmark and

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 27 The sequential read performance as shown in Figure 9 peaks at

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 28 10GbE large sequential write performance Figure 10. 10G

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 29 A single NFS client with a 10 Gigabit Ethernet to the storage

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities iii Contents Executive summary (Updated May 2012) ...

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 30 InfiniBand random write performance Figure 13. The NSS-HA

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 31 InfiniBand random read performance Figure 14. 6.4. Metada

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 32 expected the file create and file remove performance is simil

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 33 InfiniBand file stat performance Figure 16. InfiniBand f

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 34 6.5. NFSv3 compared to NFSv4 During the design and analysis

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 35 With IPoIB, sequential reads were found to be up to 22% bette

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 36 operations, NFSv3 versus NFsv4 performance depended on the nu

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 37 References 8.1) XFS: A high-performance journaling file sys

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 38 Appendix A: NSS-HA Recipe (Updated May 2012) Contents A.1. P

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 39 A.1. Pre-install preparation The following figure shows an N

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities iv A.3.2. Configure Multipath...

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 40 A.1.1. NSS-HA cluster specification The section gives a detai

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 41 PDU 2 Login name: apc password: apc port 2 for active, por

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 42 nssha61_single.py Configuring storage devices, including crea

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 43 A.2. Server hardware setup 1. Prepare two PowerEdge R710 ser

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 44 A.2.1. Checklist Before moving to the next section, please ma

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 45 A.3. Server software configuration All the operations below

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 46 A.3.2. Configure Multipath On each R710, exclude the local di

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 47 Then, edit the file /etc/multipath.conf, search for the black

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 48 A.3.3. Install Mellanox OFED package and set network IPs 1.

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 49 A.3.4. Install Operating system and storage management tools

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities v Figures Figure 1. Overview of the NSS-HA solution ...

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 50 A.3.5. Network security setting In this step ports will be en

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 51 LOCKD_TCPPORT=890 LOCKD_UDPPORT=890 MOUNTD_PORT=892 STATD_POR

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 52 2. Configure two APC PDUs, and make sure the login name, pas

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 53 A.3.8. Checklist Before moving to the next section, please ma

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 54 A.4. Performance tuning on the server All the operations bel

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 55 Restart the NFS service, service nfs restart Run the nfsstat

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 56 A.5. Storage hardware setup 1. Cable the MD3200 to the SAS 6

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 57  Cabling for 288TB configuration SAS INEMM 1EMM 2SAS OUTSAS

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 58 Note: Do not cable storage to R710 before OS install, and us

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 59 A.6. Storage configuration 1. Launch the MDSM management GU

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 6 Executive summary (Updated May 2012) This solution guide descr

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 60 Make sure the host port identifiers on each R710 match the sa

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 61 A.6.1. Checklist Before moving to the next section, please ma

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 62 A.7. NSS HA cluster setup In this recipe the term “cluster”

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 63 6. Check that the public interface is up on both servers. Th

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 64 1. Generate cluster configuration file. On the „active‟ R710

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 65 ------ ---- ---- ------ active

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 66 [root@active ~]# clustat Cluster Status for NSS61 @ Thu Jan

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 67 An SELinux policy can be generated from logs of denied operat

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 68 A.8. Quick test of HA setup 1. Test fencing: o First disabl

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 69 A.9. Useful commands and references This section provides sev

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 7 Introduction 1.This Solution Guide provides information on

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 70 3. Start cluster service. Assume the service name is HA1, it

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 71 # service cman stop A.9.5. Configure the shared storage array

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 72 size=27T features='3 queue_if_no_path pg_init_retries 50

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 73 a. Configure 144TB storage array. # vgcreate VGMD1 /dev/mapp

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 74 A.10. Performance tuning on clients 1. If the clients acce

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 75 A.11. Scripts 1. /root/config_cluster.sh file for automati

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 76 Appendix B: Benchmarks and test tools The iozone benchmark wa

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 77 The following table describes the IOZone command line argumen

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 78 IOzone IOPs Random Access (Reads and Writes) # /usr/sbin/ioz

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 79 As with the IOzone random access patterns, the following proc

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 8 Overview of the NSS-HA solution Figure 1.

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 80 For comparison, here is an example of a failing test with dat

Dell HPC NFS Storage Solution High Availability Configurations with Large Capacities 9 2.1. Availability in NSS-HA A major goal of the NSS-HA soluti