Overview of the Gfarm filesystem About the Gfarm filesystem =========================== The Gfarm filesystem is a distributed filesystem consisting of the local storage of commodity PCs. Many PCs in a local area network, compute nodes in a single cluster, multiple clusters in wide area, comprise a large-scale, high-performance shared network filesystem. The Gfarm filesystem solves performance and reliability problems in NFS and AFS by means of multiple file replicas. It not only prevents performance degradation due to access concentration, but also supports fault tolerance and disaster recovery. A unique feature of Gfarm is that each filesystem node is also a client of the Gfarm filesystem. Distributed access by filesystem nodes realizes super-scalable I/O performance. For detailed information about the Grid Datafarm architecture and preliminary performance evaluations, refer to the following paper. [1] Osamu Tatebe, Youhei Morita, Satoshi Matsuoka, Noriyuki Soda, Satoshi Sekiguchi, "Grid Datafarm Architecture for Petascale Data Intensive Computing," Proceedings of the 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGrid 2002), IEEE, pp.102-110, 2002 http://datafarm.apgrid.org/ How to access Gfarm =================== There are three methods that can be used to access the Gfarm filesystem: - Using Gfarm commands and Gfarm native file I/O APIs You can use Gfarm specific features like file replication, new file view semantics, etc., via this method. The pathname syntax of this method is gfarm:/user/dir/file. - Using the syscall hook library You can use your existing programs without any modification via this method, as if the Gfarm filesystem were mounted at /gfarm. However, there are a few restrictions. For example, you cannot use the mmap(2) feature for a file which doesn't have a replica in the local node. The pathname syntax of this method is /gfarm/user/dir/file. You can use the README.hook.en file as a description of this feature. - Using GfarmFS-FUSE You can actually mount the Gfarm filesystem from Linux clients by using FUSE (http://fuse.sourceforge.net/). Unlike the other methods, this one is completely transparent from your application. To use this feature, you have to install GfarmFS-FUSE from the following URL, in addition to the Gfarm software. http://datafarm.apgrid.org/software/gfarmfs-fuse.en.html Host types that make up the Gfarm system ======================================== A Gfarm system consists of the following kinds of nodes: - Client node A terminal node for users. - Filesystem node Filesystem nodes provide data storage and CPUs for the Gfarm system. On each filesystem node, the Gfarm filesystem daemon, called gfsd, is running to facilitate remote file operations and access control in the Gfarm filesystem, as well as to provide user authentication, file replication, fast invocation, node resource status monitoring, and control. - Metadata server node A metadata server node manages Gfarm filesystem metadata and parallel process information. On the metadata server node, a Gfarm filesystem metaserver (gfmd), and a backend database server such as an LDAP server (slapd) or a PostgreSQL server (postmaster) are running. - Metadata cache server node A metadata cache server is a proxy server to make metadata access faster, and to prevent metadata access concentration. On metadata cache server nodes, gfarm_agent is running. The four types of nodes just introduced are not necessarily different hosts, i.e., you can use the same host for the above purposes, if the number of available hosts are limited. A file in the Gfarm filesystem is a standard file or a large-scale file that is divided into fragments or into a ranked group of files. Physically, each file is replicated and dispersed across the disks of the filesystem nodes, and they will be accessed in parallel. Structure of the Gfarm software =============================== The Gfarm filesystem consists of the following software: - The libgfarm.a library A library that implements Gfarm APIs, including Gfarm file access, file replication, parallel I/O extensions, parallel file transfer, and file-affinity process scheduling. Parallel I/O extensions provide new file views; an index file view and a local file view, for parallel processes. - PostgreSQL server or OpenLDAP server One of these runs on a Gfarm filesystem metadata server to provide metadata information, e.g., where the contents of files are stored, etc. - gfmd - the Gfarm filesystem metadata server gfmd must be running on a metadata server node in advance. In the current implementation, gfmd only manages information on running parallel processes which are invoked by the gfrun command. You can use the gfps command to acquire information registered in gfmd. - gfarm_agent - the Gfarm metadata cache server gfarm_agent is a proxy server for the filesystem metadata server. It improves metadata access performance. Typically, it is needed for each PC cluster. You can use the gfhost command to register the metadata of the filesystem nodes, and to see the status of gfsd. - gfsd - the Gfarm filesystem daemon An I/O daemon for the Gfarm filesystem that runs on every filesystem node, which provides remote file operations with access control, as well as user authentication, file replication, fast invocation, and node resource status monitoring. - Gfarm command tools Gfarm command tools consist of filesystem commands such as gfls, gfrm, gfwhere and gfrep; a filesystem node management tool, gfhost; file management tools such as gfreg and gfexport; session key management tools, such as gfkey; and parallel process management tools, such as gfps, gfrun and gfmpirun_p4. There are Gfarm parallel commands: gfcp for a parallel copy, gfgrep for a parallel grep, gfwc for a parallel word-count, etc. About authentication ==================== gfmd and gfsd support the following three authentication methods. Please read the security section in Gfarm-FAQ.en, too. 1. sharedsecret This uses a shared key in the ~/.gfarm_shared_key file which will be generated automatically by the Gfarm software. This is suitable for an environment that is protected by a firewall. This authentication method is easy to use in an environment which shares users' home directories via NFS. 2. gsi This is the GSI -- Grid Security Infrastructure -- method, and it uses public key authentication, which is based on a PKI-style certificate. This method encrypts network communication, and is suitable for use over the Internet. Please read the following page provided by the Globus project for details: http://www.globus.org/security/overview.html 3. gsi_auth This method uses GSI for its authentication, but switches to a plain TCP connection after the authentication is completed. This method is suitable for an environment that is protected by a firewall. PostgreSQL and OpenLDAP use their own authentication methods. The current version of gfarm_agent doesn't use any authentication, thus, it must be located on hosts which are protected by a firewall.