1.. SPDX-License-Identifier: GPL-2.0
2
3========
4ORANGEFS
5========
6
7OrangeFS is an LGPL userspace scale-out parallel storage system. It is ideal
8for large storage problems faced by HPC, BigData, Streaming Video,
9Genomics, Bioinformatics.
10
11Orangefs, originally called PVFS, was first developed in 1993 by
12Walt Ligon and Eric Blumer as a parallel file system for Parallel
13Virtual Machine (PVM) as part of a NASA grant to study the I/O patterns
14of parallel programs.
15
16Orangefs features include:
17
18  * Distributes file data among multiple file servers
19  * Supports simultaneous access by multiple clients
20  * Stores file data and metadata on servers using local file system
21    and access methods
22  * Userspace implementation is easy to install and maintain
23  * Direct MPI support
24  * Stateless
25
26
27Mailing List Archives
28=====================
29
30http://lists.orangefs.org/pipermail/devel_lists.orangefs.org/
31
32
33Mailing List Submissions
34========================
35
36devel@lists.orangefs.org
37
38
39Documentation
40=============
41
42http://www.orangefs.org/documentation/
43
44Running ORANGEFS On a Single Server
45===================================
46
47OrangeFS is usually run in large installations with multiple servers and
48clients, but a complete filesystem can be run on a single machine for
49development and testing.
50
51On Fedora, install orangefs and orangefs-server::
52
53    dnf -y install orangefs orangefs-server
54
55There is an example server configuration file in
56/etc/orangefs/orangefs.conf.  Change localhost to your hostname if
57necessary.
58
59To generate a filesystem to run xfstests against, see below.
60
61There is an example client configuration file in /etc/pvfs2tab.  It is a
62single line.  Uncomment it and change the hostname if necessary.  This
63controls clients which use libpvfs2.  This does not control the
64pvfs2-client-core.
65
66Create the filesystem::
67
68    pvfs2-server -f /etc/orangefs/orangefs.conf
69
70Start the server::
71
72    systemctl start orangefs-server
73
74Test the server::
75
76    pvfs2-ping -m /pvfsmnt
77
78Start the client.  The module must be compiled in or loaded before this
79point::
80
81    systemctl start orangefs-client
82
83Mount the filesystem::
84
85    mount -t pvfs2 tcp://localhost:3334/orangefs /pvfsmnt
86
87Userspace Filesystem Source
88===========================
89
90http://www.orangefs.org/download
91
92Orangefs versions prior to 2.9.3 would not be compatible with the
93upstream version of the kernel client.
94
95
96Building ORANGEFS on a Single Server
97====================================
98
99Where OrangeFS cannot be installed from distribution packages, it may be
100built from source.
101
102You can omit --prefix if you don't care that things are sprinkled around
103in /usr/local.  As of version 2.9.6, OrangeFS uses Berkeley DB by
104default, we will probably be changing the default to LMDB soon.
105
106::
107
108    ./configure --prefix=/opt/ofs --with-db-backend=lmdb --disable-usrint
109
110    make
111
112    make install
113
114Create an orangefs config file by running pvfs2-genconfig and
115specifying a target config file. Pvfs2-genconfig will prompt you
116through. Generally it works fine to take the defaults, but you
117should use your server's hostname, rather than "localhost" when
118it comes to that question::
119
120    /opt/ofs/bin/pvfs2-genconfig /etc/pvfs2.conf
121
122Create an /etc/pvfs2tab file::
123
124Localhost is fine for your pvfs2tab file:
125
126    echo tcp://localhost:3334/orangefs /pvfsmnt pvfs2 defaults,noauto 0 0 > \
127	/etc/pvfs2tab
128
129Create the mount point you specified in the tab file if needed::
130
131    mkdir /pvfsmnt
132
133Bootstrap the server::
134
135    /opt/ofs/sbin/pvfs2-server -f /etc/pvfs2.conf
136
137Start the server::
138
139    /opt/ofs/sbin/pvfs2-server /etc/pvfs2.conf
140
141Now the server should be running. Pvfs2-ls is a simple
142test to verify that the server is running::
143
144    /opt/ofs/bin/pvfs2-ls /pvfsmnt
145
146If stuff seems to be working, load the kernel module and
147turn on the client core::
148
149    /opt/ofs/sbin/pvfs2-client -p /opt/ofs/sbin/pvfs2-client-core
150
151Mount your filesystem::
152
153    mount -t pvfs2 tcp://`hostname`:3334/orangefs /pvfsmnt
154
155
156Running xfstests
157================
158
159It is useful to use a scratch filesystem with xfstests.  This can be
160done with only one server.
161
162Make a second copy of the FileSystem section in the server configuration
163file, which is /etc/orangefs/orangefs.conf.  Change the Name to scratch.
164Change the ID to something other than the ID of the first FileSystem
165section (2 is usually a good choice).
166
167Then there are two FileSystem sections: orangefs and scratch.
168
169This change should be made before creating the filesystem.
170
171::
172
173    pvfs2-server -f /etc/orangefs/orangefs.conf
174
175To run xfstests, create /etc/xfsqa.config::
176
177    TEST_DIR=/orangefs
178    TEST_DEV=tcp://localhost:3334/orangefs
179    SCRATCH_MNT=/scratch
180    SCRATCH_DEV=tcp://localhost:3334/scratch
181
182Then xfstests can be run::
183
184    ./check -pvfs2
185
186
187Options
188=======
189
190The following mount options are accepted:
191
192  acl
193    Allow the use of Access Control Lists on files and directories.
194
195  intr
196    Some operations between the kernel client and the user space
197    filesystem can be interruptible, such as changes in debug levels
198    and the setting of tunable parameters.
199
200  local_lock
201    Enable posix locking from the perspective of "this" kernel. The
202    default file_operations lock action is to return ENOSYS. Posix
203    locking kicks in if the filesystem is mounted with -o local_lock.
204    Distributed locking is being worked on for the future.
205
206
207Debugging
208=========
209
210If you want the debug (GOSSIP) statements in a particular
211source file (inode.c for example) go to syslog::
212
213  echo inode > /sys/kernel/debug/orangefs/kernel-debug
214
215No debugging (the default)::
216
217  echo none > /sys/kernel/debug/orangefs/kernel-debug
218
219Debugging from several source files::
220
221  echo inode,dir > /sys/kernel/debug/orangefs/kernel-debug
222
223All debugging::
224
225  echo all > /sys/kernel/debug/orangefs/kernel-debug
226
227Get a list of all debugging keywords::
228
229  cat /sys/kernel/debug/orangefs/debug-help
230
231
232Protocol between Kernel Module and Userspace
233============================================
234
235Orangefs is a user space filesystem and an associated kernel module.
236We'll just refer to the user space part of Orangefs as "userspace"
237from here on out. Orangefs descends from PVFS, and userspace code
238still uses PVFS for function and variable names. Userspace typedefs
239many of the important structures. Function and variable names in
240the kernel module have been transitioned to "orangefs", and The Linux
241Coding Style avoids typedefs, so kernel module structures that
242correspond to userspace structures are not typedefed.
243
244The kernel module implements a pseudo device that userspace
245can read from and write to. Userspace can also manipulate the
246kernel module through the pseudo device with ioctl.
247
248The Bufmap
249----------
250
251At startup userspace allocates two page-size-aligned (posix_memalign)
252mlocked memory buffers, one is used for IO and one is used for readdir
253operations. The IO buffer is 41943040 bytes and the readdir buffer is
2544194304 bytes. Each buffer contains logical chunks, or partitions, and
255a pointer to each buffer is added to its own PVFS_dev_map_desc structure
256which also describes its total size, as well as the size and number of
257the partitions.
258
259A pointer to the IO buffer's PVFS_dev_map_desc structure is sent to a
260mapping routine in the kernel module with an ioctl. The structure is
261copied from user space to kernel space with copy_from_user and is used
262to initialize the kernel module's "bufmap" (struct orangefs_bufmap), which
263then contains:
264
265  * refcnt
266    - a reference counter
267  * desc_size - PVFS2_BUFMAP_DEFAULT_DESC_SIZE (4194304) - the IO buffer's
268    partition size, which represents the filesystem's block size and
269    is used for s_blocksize in super blocks.
270  * desc_count - PVFS2_BUFMAP_DEFAULT_DESC_COUNT (10) - the number of
271    partitions in the IO buffer.
272  * desc_shift - log2(desc_size), used for s_blocksize_bits in super blocks.
273  * total_size - the total size of the IO buffer.
274  * page_count - the number of 4096 byte pages in the IO buffer.
275  * page_array - a pointer to ``page_count * (sizeof(struct page*))`` bytes
276    of kcalloced memory. This memory is used as an array of pointers
277    to each of the pages in the IO buffer through a call to get_user_pages.
278  * desc_array - a pointer to ``desc_count * (sizeof(struct orangefs_bufmap_desc))``
279    bytes of kcalloced memory. This memory is further intialized:
280
281      user_desc is the kernel's copy of the IO buffer's ORANGEFS_dev_map_desc
282      structure. user_desc->ptr points to the IO buffer.
283
284      ::
285
286	pages_per_desc = bufmap->desc_size / PAGE_SIZE
287	offset = 0
288
289        bufmap->desc_array[0].page_array = &bufmap->page_array[offset]
290        bufmap->desc_array[0].array_count = pages_per_desc = 1024
291        bufmap->desc_array[0].uaddr = (user_desc->ptr) + (0 * 1024 * 4096)
292        offset += 1024
293                           .
294                           .
295                           .
296        bufmap->desc_array[9].page_array = &bufmap->page_array[offset]
297        bufmap->desc_array[9].array_count = pages_per_desc = 1024
298        bufmap->desc_array[9].uaddr = (user_desc->ptr) +
299                                               (9 * 1024 * 4096)
300        offset += 1024
301
302  * buffer_index_array - a desc_count sized array of ints, used to
303    indicate which of the IO buffer's partitions are available to use.
304  * buffer_index_lock - a spinlock to protect buffer_index_array during update.
305  * readdir_index_array - a five (ORANGEFS_READDIR_DEFAULT_DESC_COUNT) element
306    int array used to indicate which of the readdir buffer's partitions are
307    available to use.
308  * readdir_index_lock - a spinlock to protect readdir_index_array during
309    update.
310
311Operations
312----------
313
314The kernel module builds an "op" (struct orangefs_kernel_op_s) when it
315needs to communicate with userspace. Part of the op contains the "upcall"
316which expresses the request to userspace. Part of the op eventually
317contains the "downcall" which expresses the results of the request.
318
319The slab allocator is used to keep a cache of op structures handy.
320
321At init time the kernel module defines and initializes a request list
322and an in_progress hash table to keep track of all the ops that are
323in flight at any given time.
324
325Ops are stateful:
326
327 * unknown
328	    - op was just initialized
329 * waiting
330	    - op is on request_list (upward bound)
331 * inprogr
332	    - op is in progress (waiting for downcall)
333 * serviced
334	    - op has matching downcall; ok
335 * purged
336	    - op has to start a timer since client-core
337              exited uncleanly before servicing op
338 * given up
339	    - submitter has given up waiting for it
340
341When some arbitrary userspace program needs to perform a
342filesystem operation on Orangefs (readdir, I/O, create, whatever)
343an op structure is initialized and tagged with a distinguishing ID
344number. The upcall part of the op is filled out, and the op is
345passed to the "service_operation" function.
346
347Service_operation changes the op's state to "waiting", puts
348it on the request list, and signals the Orangefs file_operations.poll
349function through a wait queue. Userspace is polling the pseudo-device
350and thus becomes aware of the upcall request that needs to be read.
351
352When the Orangefs file_operations.read function is triggered, the
353request list is searched for an op that seems ready-to-process.
354The op is removed from the request list. The tag from the op and
355the filled-out upcall struct are copy_to_user'ed back to userspace.
356
357If any of these (and some additional protocol) copy_to_users fail,
358the op's state is set to "waiting" and the op is added back to
359the request list. Otherwise, the op's state is changed to "in progress",
360and the op is hashed on its tag and put onto the end of a list in the
361in_progress hash table at the index the tag hashed to.
362
363When userspace has assembled the response to the upcall, it
364writes the response, which includes the distinguishing tag, back to
365the pseudo device in a series of io_vecs. This triggers the Orangefs
366file_operations.write_iter function to find the op with the associated
367tag and remove it from the in_progress hash table. As long as the op's
368state is not "canceled" or "given up", its state is set to "serviced".
369The file_operations.write_iter function returns to the waiting vfs,
370and back to service_operation through wait_for_matching_downcall.
371
372Service operation returns to its caller with the op's downcall
373part (the response to the upcall) filled out.
374
375The "client-core" is the bridge between the kernel module and
376userspace. The client-core is a daemon. The client-core has an
377associated watchdog daemon. If the client-core is ever signaled
378to die, the watchdog daemon restarts the client-core. Even though
379the client-core is restarted "right away", there is a period of
380time during such an event that the client-core is dead. A dead client-core
381can't be triggered by the Orangefs file_operations.poll function.
382Ops that pass through service_operation during a "dead spell" can timeout
383on the wait queue and one attempt is made to recycle them. Obviously,
384if the client-core stays dead too long, the arbitrary userspace processes
385trying to use Orangefs will be negatively affected. Waiting ops
386that can't be serviced will be removed from the request list and
387have their states set to "given up". In-progress ops that can't
388be serviced will be removed from the in_progress hash table and
389have their states set to "given up".
390
391Readdir and I/O ops are atypical with respect to their payloads.
392
393  - readdir ops use the smaller of the two pre-allocated pre-partitioned
394    memory buffers. The readdir buffer is only available to userspace.
395    The kernel module obtains an index to a free partition before launching
396    a readdir op. Userspace deposits the results into the indexed partition
397    and then writes them to back to the pvfs device.
398
399  - io (read and write) ops use the larger of the two pre-allocated
400    pre-partitioned memory buffers. The IO buffer is accessible from
401    both userspace and the kernel module. The kernel module obtains an
402    index to a free partition before launching an io op. The kernel module
403    deposits write data into the indexed partition, to be consumed
404    directly by userspace. Userspace deposits the results of read
405    requests into the indexed partition, to be consumed directly
406    by the kernel module.
407
408Responses to kernel requests are all packaged in pvfs2_downcall_t
409structs. Besides a few other members, pvfs2_downcall_t contains a
410union of structs, each of which is associated with a particular
411response type.
412
413The several members outside of the union are:
414
415 ``int32_t type``
416    - type of operation.
417 ``int32_t status``
418    - return code for the operation.
419 ``int64_t trailer_size``
420    - 0 unless readdir operation.
421 ``char *trailer_buf``
422    - initialized to NULL, used during readdir operations.
423
424The appropriate member inside the union is filled out for any
425particular response.
426
427  PVFS2_VFS_OP_FILE_IO
428    fill a pvfs2_io_response_t
429
430  PVFS2_VFS_OP_LOOKUP
431    fill a PVFS_object_kref
432
433  PVFS2_VFS_OP_CREATE
434    fill a PVFS_object_kref
435
436  PVFS2_VFS_OP_SYMLINK
437    fill a PVFS_object_kref
438
439  PVFS2_VFS_OP_GETATTR
440    fill in a PVFS_sys_attr_s (tons of stuff the kernel doesn't need)
441    fill in a string with the link target when the object is a symlink.
442
443  PVFS2_VFS_OP_MKDIR
444    fill a PVFS_object_kref
445
446  PVFS2_VFS_OP_STATFS
447    fill a pvfs2_statfs_response_t with useless info <g>. It is hard for
448    us to know, in a timely fashion, these statistics about our
449    distributed network filesystem.
450
451  PVFS2_VFS_OP_FS_MOUNT
452    fill a pvfs2_fs_mount_response_t which is just like a PVFS_object_kref
453    except its members are in a different order and "__pad1" is replaced
454    with "id".
455
456  PVFS2_VFS_OP_GETXATTR
457    fill a pvfs2_getxattr_response_t
458
459  PVFS2_VFS_OP_LISTXATTR
460    fill a pvfs2_listxattr_response_t
461
462  PVFS2_VFS_OP_PARAM
463    fill a pvfs2_param_response_t
464
465  PVFS2_VFS_OP_PERF_COUNT
466    fill a pvfs2_perf_count_response_t
467
468  PVFS2_VFS_OP_FSKEY
469    file a pvfs2_fs_key_response_t
470
471  PVFS2_VFS_OP_READDIR
472    jamb everything needed to represent a pvfs2_readdir_response_t into
473    the readdir buffer descriptor specified in the upcall.
474
475Userspace uses writev() on /dev/pvfs2-req to pass responses to the requests
476made by the kernel side.
477
478A buffer_list containing:
479
480  - a pointer to the prepared response to the request from the
481    kernel (struct pvfs2_downcall_t).
482  - and also, in the case of a readdir request, a pointer to a
483    buffer containing descriptors for the objects in the target
484    directory.
485
486... is sent to the function (PINT_dev_write_list) which performs
487the writev.
488
489PINT_dev_write_list has a local iovec array: struct iovec io_array[10];
490
491The first four elements of io_array are initialized like this for all
492responses::
493
494  io_array[0].iov_base = address of local variable "proto_ver" (int32_t)
495  io_array[0].iov_len = sizeof(int32_t)
496
497  io_array[1].iov_base = address of global variable "pdev_magic" (int32_t)
498  io_array[1].iov_len = sizeof(int32_t)
499
500  io_array[2].iov_base = address of parameter "tag" (PVFS_id_gen_t)
501  io_array[2].iov_len = sizeof(int64_t)
502
503  io_array[3].iov_base = address of out_downcall member (pvfs2_downcall_t)
504                         of global variable vfs_request (vfs_request_t)
505  io_array[3].iov_len = sizeof(pvfs2_downcall_t)
506
507Readdir responses initialize the fifth element io_array like this::
508
509  io_array[4].iov_base = contents of member trailer_buf (char *)
510                         from out_downcall member of global variable
511                         vfs_request
512  io_array[4].iov_len = contents of member trailer_size (PVFS_size)
513                        from out_downcall member of global variable
514                        vfs_request
515
516Orangefs exploits the dcache in order to avoid sending redundant
517requests to userspace. We keep object inode attributes up-to-date with
518orangefs_inode_getattr. Orangefs_inode_getattr uses two arguments to
519help it decide whether or not to update an inode: "new" and "bypass".
520Orangefs keeps private data in an object's inode that includes a short
521timeout value, getattr_time, which allows any iteration of
522orangefs_inode_getattr to know how long it has been since the inode was
523updated. When the object is not new (new == 0) and the bypass flag is not
524set (bypass == 0) orangefs_inode_getattr returns without updating the inode
525if getattr_time has not timed out. Getattr_time is updated each time the
526inode is updated.
527
528Creation of a new object (file, dir, sym-link) includes the evaluation of
529its pathname, resulting in a negative directory entry for the object.
530A new inode is allocated and associated with the dentry, turning it from
531a negative dentry into a "productive full member of society". Orangefs
532obtains the new inode from Linux with new_inode() and associates
533the inode with the dentry by sending the pair back to Linux with
534d_instantiate().
535
536The evaluation of a pathname for an object resolves to its corresponding
537dentry. If there is no corresponding dentry, one is created for it in
538the dcache. Whenever a dentry is modified or verified Orangefs stores a
539short timeout value in the dentry's d_time, and the dentry will be trusted
540for that amount of time. Orangefs is a network filesystem, and objects
541can potentially change out-of-band with any particular Orangefs kernel module
542instance, so trusting a dentry is risky. The alternative to trusting
543dentries is to always obtain the needed information from userspace - at
544least a trip to the client-core, maybe to the servers. Obtaining information
545from a dentry is cheap, obtaining it from userspace is relatively expensive,
546hence the motivation to use the dentry when possible.
547
548The timeout values d_time and getattr_time are jiffy based, and the
549code is designed to avoid the jiffy-wrap problem::
550
551    "In general, if the clock may have wrapped around more than once, there
552    is no way to tell how much time has elapsed. However, if the times t1
553    and t2 are known to be fairly close, we can reliably compute the
554    difference in a way that takes into account the possibility that the
555    clock may have wrapped between times."
556
557from course notes by instructor Andy Wang
558
559