1===================
2Setting up NFS/RDMA
3===================
4
5:Author:
6  NetApp and Open Grid Computing (May 29, 2008)
7
8.. warning::
9  This document is probably obsolete.
10
11Overview
12========
13
14This document describes how to install and setup the Linux NFS/RDMA client
15and server software.
16
17The NFS/RDMA client was first included in Linux 2.6.24. The NFS/RDMA server
18was first included in the following release, Linux 2.6.25.
19
20In our testing, we have obtained excellent performance results (full 10Gbit
21wire bandwidth at minimal client CPU) under many workloads. The code passes
22the full Connectathon test suite and operates over both Infiniband and iWARP
23RDMA adapters.
24
25Getting Help
26============
27
28If you get stuck, you can ask questions on the
29nfs-rdma-devel@lists.sourceforge.net mailing list.
30
31Installation
32============
33
34These instructions are a step by step guide to building a machine for
35use with NFS/RDMA.
36
37- Install an RDMA device
38
39  Any device supported by the drivers in drivers/infiniband/hw is acceptable.
40
41  Testing has been performed using several Mellanox-based IB cards, the
42  Ammasso AMS1100 iWARP adapter, and the Chelsio cxgb3 iWARP adapter.
43
44- Install a Linux distribution and tools
45
46  The first kernel release to contain both the NFS/RDMA client and server was
47  Linux 2.6.25  Therefore, a distribution compatible with this and subsequent
48  Linux kernel release should be installed.
49
50  The procedures described in this document have been tested with
51  distributions from Red Hat's Fedora Project (http://fedora.redhat.com/).
52
53- Install nfs-utils-1.1.2 or greater on the client
54
55  An NFS/RDMA mount point can be obtained by using the mount.nfs command in
56  nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils
57  version with support for NFS/RDMA mounts, but for various reasons we
58  recommend using nfs-utils-1.1.2 or greater). To see which version of
59  mount.nfs you are using, type:
60
61  .. code-block:: sh
62
63    $ /sbin/mount.nfs -V
64
65  If the version is less than 1.1.2 or the command does not exist,
66  you should install the latest version of nfs-utils.
67
68  Download the latest package from: https://www.kernel.org/pub/linux/utils/nfs
69
70  Uncompress the package and follow the installation instructions.
71
72  If you will not need the idmapper and gssd executables (you do not need
73  these to create an NFS/RDMA enabled mount command), the installation
74  process can be simplified by disabling these features when running
75  configure:
76
77  .. code-block:: sh
78
79    $ ./configure --disable-gss --disable-nfsv4
80
81  To build nfs-utils you will need the tcp_wrappers package installed. For
82  more information on this see the package's README and INSTALL files.
83
84  After building the nfs-utils package, there will be a mount.nfs binary in
85  the utils/mount directory. This binary can be used to initiate NFS v2, v3,
86  or v4 mounts. To initiate a v4 mount, the binary must be called
87  mount.nfs4.  The standard technique is to create a symlink called
88  mount.nfs4 to mount.nfs.
89
90  This mount.nfs binary should be installed at /sbin/mount.nfs as follows:
91
92  .. code-block:: sh
93
94    $ sudo cp utils/mount/mount.nfs /sbin/mount.nfs
95
96  In this location, mount.nfs will be invoked automatically for NFS mounts
97  by the system mount command.
98
99    .. note::
100      mount.nfs and therefore nfs-utils-1.1.2 or greater is only needed
101      on the NFS client machine. You do not need this specific version of
102      nfs-utils on the server. Furthermore, only the mount.nfs command from
103      nfs-utils-1.1.2 is needed on the client.
104
105- Install a Linux kernel with NFS/RDMA
106
107  The NFS/RDMA client and server are both included in the mainline Linux
108  kernel version 2.6.25 and later. This and other versions of the Linux
109  kernel can be found at: https://www.kernel.org/pub/linux/kernel/
110
111  Download the sources and place them in an appropriate location.
112
113- Configure the RDMA stack
114
115  Make sure your kernel configuration has RDMA support enabled. Under
116  Device Drivers -> InfiniBand support, update the kernel configuration
117  to enable InfiniBand support [NOTE: the option name is misleading. Enabling
118  InfiniBand support is required for all RDMA devices (IB, iWARP, etc.)].
119
120  Enable the appropriate IB HCA support (mlx4, mthca, ehca, ipath, etc.) or
121  iWARP adapter support (amso, cxgb3, etc.).
122
123  If you are using InfiniBand, be sure to enable IP-over-InfiniBand support.
124
125- Configure the NFS client and server
126
127  Your kernel configuration must also have NFS file system support and/or
128  NFS server support enabled. These and other NFS related configuration
129  options can be found under File Systems -> Network File Systems.
130
131- Build, install, reboot
132
133  The NFS/RDMA code will be enabled automatically if NFS and RDMA
134  are turned on. The NFS/RDMA client and server are configured via the hidden
135  SUNRPC_XPRT_RDMA config option that depends on SUNRPC and INFINIBAND. The
136  value of SUNRPC_XPRT_RDMA will be:
137
138    #. N if either SUNRPC or INFINIBAND are N, in this case the NFS/RDMA client
139       and server will not be built
140
141    #. M if both SUNRPC and INFINIBAND are on (M or Y) and at least one is M,
142       in this case the NFS/RDMA client and server will be built as modules
143
144    #. Y if both SUNRPC and INFINIBAND are Y, in this case the NFS/RDMA client
145       and server will be built into the kernel
146
147  Therefore, if you have followed the steps above and turned no NFS and RDMA,
148  the NFS/RDMA client and server will be built.
149
150  Build a new kernel, install it, boot it.
151
152Check RDMA and NFS Setup
153========================
154
155Before configuring the NFS/RDMA software, it is a good idea to test
156your new kernel to ensure that the kernel is working correctly.
157In particular, it is a good idea to verify that the RDMA stack
158is functioning as expected and standard NFS over TCP/IP and/or UDP/IP
159is working properly.
160
161- Check RDMA Setup
162
163  If you built the RDMA components as modules, load them at
164  this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel
165  card:
166
167  .. code-block:: sh
168
169    $ modprobe ib_mthca
170    $ modprobe ib_ipoib
171
172  If you are using InfiniBand, make sure there is a Subnet Manager (SM)
173  running on the network. If your IB switch has an embedded SM, you can
174  use it. Otherwise, you will need to run an SM, such as OpenSM, on one
175  of your end nodes.
176
177  If an SM is running on your network, you should see the following:
178
179  .. code-block:: sh
180
181    $ cat /sys/class/infiniband/driverX/ports/1/state
182    4: ACTIVE
183
184  where driverX is mthca0, ipath5, ehca3, etc.
185
186  To further test the InfiniBand software stack, use IPoIB (this
187  assumes you have two IB hosts named host1 and host2):
188
189  .. code-block:: sh
190
191    host1$ ip link set dev ib0 up
192    host1$ ip address add dev ib0 a.b.c.x
193    host2$ ip link set dev ib0 up
194    host2$ ip address add dev ib0 a.b.c.y
195    host1$ ping a.b.c.y
196    host2$ ping a.b.c.x
197
198  For other device types, follow the appropriate procedures.
199
200- Check NFS Setup
201
202  For the NFS components enabled above (client and/or server),
203  test their functionality over standard Ethernet using TCP/IP or UDP/IP.
204
205NFS/RDMA Setup
206==============
207
208We recommend that you use two machines, one to act as the client and
209one to act as the server.
210
211One time configuration:
212-----------------------
213
214- On the server system, configure the /etc/exports file and start the NFS/RDMA server.
215
216  Exports entries with the following formats have been tested::
217
218  /vol0   192.168.0.47(fsid=0,rw,async,insecure,no_root_squash)
219  /vol0   192.168.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash)
220
221  The IP address(es) is(are) the client's IPoIB address for an InfiniBand
222  HCA or the client's iWARP address(es) for an RNIC.
223
224  .. note::
225    The "insecure" option must be used because the NFS/RDMA client does
226    not use a reserved port.
227
228Each time a machine boots:
229--------------------------
230
231- Load and configure the RDMA drivers
232
233  For InfiniBand using a Mellanox adapter:
234
235  .. code-block:: sh
236
237    $ modprobe ib_mthca
238    $ modprobe ib_ipoib
239    $ ip li set dev ib0 up
240    $ ip addr add dev ib0 a.b.c.d
241
242  .. note::
243    Please use unique addresses for the client and server!
244
245- Start the NFS server
246
247  If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in
248  kernel config), load the RDMA transport module:
249
250  .. code-block:: sh
251
252    $ modprobe svcrdma
253
254  Regardless of how the server was built (module or built-in), start the
255  server:
256
257  .. code-block:: sh
258
259    $ /etc/init.d/nfs start
260
261  or
262
263  .. code-block:: sh
264
265    $ service nfs start
266
267  Instruct the server to listen on the RDMA transport:
268
269  .. code-block:: sh
270
271    $ echo rdma 20049 > /proc/fs/nfsd/portlist
272
273- On the client system
274
275  If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in
276  kernel config), load the RDMA client module:
277
278  .. code-block:: sh
279
280    $ modprobe xprtrdma.ko
281
282  Regardless of how the client was built (module or built-in), use this
283  command to mount the NFS/RDMA server:
284
285  .. code-block:: sh
286
287    $ mount -o rdma,port=20049 <IPoIB-server-name-or-address>:/<export> /mnt
288
289  To verify that the mount is using RDMA, run "cat /proc/mounts" and check
290  the "proto" field for the given mount.
291
292  Congratulations! You're using NFS/RDMA!
293