gfarm.conf is a text file that contains a Gfarm configuration. Gfarm server processes, such as gfmd and gfsd, refer to %%SYSCONFDIR%%/gfarm.conf by default. Since this configuration file is only read at startup, it is necessary to restart servers when the contents of the configuration file are updated.
Application programs, such as gfls and gfhost, refer to both %%SYSCONFDIR%%/gfarm.conf, and a file specified by an environment variable, GFARM_CONFIG_FILE. If both configuration files exist, the file specified by the environment variable, GFARM_CONFIG_FILE, is read first. Both files have the same grammar. If the environment variable GFARM_CONFIG_FILE doesn't exist, ~/.gfarmrc in a user's home directory is used instead.
Each line of gfarm.conf consists of one statement. When the line ends with the character ``\'', the line continues for the next line. A word beginning with ``#'' causes that word and all remaining characters on that line to be ignored.
Host_specification has the following forms.
Specifies an IP address with four octets from 0 to 255, separated by ".".
Specifies a network address with an IP address and a netmask from 0 to 31 separated by "/".
Specifies a host name.
Specifies all hosts which belong to the domain.name.
Specifies all hosts.
The following statements are supported.
The spool statement specifies a spool directory for a Gfarm filesystem on this filesystem node. On the filesystem node, this statement should be specified.
For example,
spool /var/spool/gfarm
The spool_server_listen_address statement specifies the IP address at which the gfsd accepts TCP and UDP requests. The default address is all IP addresses of the host. This option is useful when one wants to invoke multiple gfsd to provide multiple spool directories on the host.
For example,
spool_server_listen_address 192.168.121.1
The spool_serverport statement specifies the tcp and udp port number the gfsd is listening on. The default port number is 600.
For example,
spool_serverport 600
This statement specifies the type of credential used by gfsd for GSI authentication. This is ignored when you are using sharedsecret authentication.
If this statement isn't used on the server side, the server uses a host certificate, if the server is invoked with root privileges. Or, if the server is invoked as a non-privileged user, the server uses the user's certificate. Thus, you usually don't need to use this statement on the server side.
If this statement isn't used on the client side, the client assumes that the server that the client is going to connect is using a host certificate of the server host. Thus, if the server is not invoked with root privileges, but invoked with user privileges where the user is the same as the user who invoked the client, the client side needs to specify the following one line.
Example:
spool_server_cred_type self
The possible types of cred_type are ``self'', ``host'', ``user'' and ``mechanism-specific''. And those are used with the spool_server_cred_service and spool_server_cred_name statements as follows:
This keyword specifies that the certificate that the user currently has is used. You must not use either the spool_server_cred_service or spool_server_cred_name statement, if you are using this type.
This keyword specifies that a host certificate or a service certificate is used. To choose a service certificate, the name of the service may be specified by the spool_server_cred_service statement. If ``host'' is specified as the service name, a host certificate in the file ``/etc/grid-security/hostcert.pem'' will be used. If any server name other than ``host'' is specified, a service certificate in the file ``/etc/grid-security/SERVICE/SERVICEcert.pem'' will be used. If the service name is omitted, ``host'' will be used as the service name by default. Only the Common Name field of a certificate will be used to check the server's identity for both a host certificate and a service certificate. And the Common Name field must be in the ``CN=SERVERNAME/HOSTNAME'' format. Also, the hostname must match the canonical name configured by the gfhost command exactly. Alias hostnames are not allowed.
This feature corresponds to the GSS_C_NT_HOSTBASED_SERVICE feature in GSSAPI (RFC2743/RFC2744).
Example:
spool_server_cred_type host
spool_server_cred_service host
This keyword specifies that a user certificate is used. The account name of the user may be specified by the spool_server_cred_name statement. If the account name is omitted, the user who invoked the command will be used by default. You must not specify a service name using the spool_server_cred_service statement, if you are using a user certificate.
Currently, there are several implementation limitations in this feature. First, the account name is interpreted as an account on the local host, but this is a bug, and will be changed to a gfarm global user name in the future. Second, to map from the account name to a Distinguished Name of a certificate, file ``/etc/grid-security/grid-mapfile'' is used. Thus, if there isn't such a file, or if the user isn't specified in this file, this feature cannot be used. Third, if there are multiple Distinguished Names per one account in this file, only the first Distinguished Name is used, and the rest of the Distinguished Names for the account are ignored.
This feature corresponds to the GSS_C_NT_USER_NAME feature in GSSAPI (RFC2743/RFC2744).
Example:
spool_server_cred_type user
spool_server_cred_name guest
This keyword specifies that spool_server_cred_name is treated as a raw X.509 Distinguished Name serving as a server's certificate. You must not specify a service name using a spool_server_cred_service statement, if you are using this type.
This feature corresponds to a case where GSS_C_NO_OID is specified as a Name Type in GSSAPI (RFC2743/RFC2744).
Example:
spool_server_cred_type mechanism-specific
spool_server_cred_name "/O=Grid/O=Globus/OU=example.com/CN=John Smith"
This statement specifies the service name of a service certificate used by gfsd for GSI authentication, when ``host'' is specified in spool_server_cred_type statement. This is ignored when you are using sharedsecret authentication. Please read the description of the spool_server_cred_type statement for details.
This statement specifies the setting of a certificate used by gfsd for GSI authentication. What this setting means depends on the type specified in the spool_server_cred_type statement. This is ignored when you are using sharedsecret authentication. Please read the description of the spool_server_cred_type statement for details.
The metadb_serverhost statement specifies the host name on which gfmd is running.
This statement cannot be omitted.
For example,
metadb_serverhost ldap.example.com
The metadb_serverport statement specifies the tcp port number the gfmd is listening on. The default port number is 601.
For example,
metadb_serverport 601
This statement specifies the type of credential used by gfmd for GSI authentication. This is ignored when you are using sharedsecret authentication. Please read the description of the spool_server_cred_type statement on the configuration of this statement.
This statement specifies the service name of a service certificate used by gfmd for GSI authentication, when ``host'' is specified in metadb_server_cred_type statement. This is ignored when you are using sharedsecret authentication. Please read the description of the spool_server_cred_type statement on the configuration of this statement.
This statement specifies the setting of a certificate used by gfmd for GSI authentication. What this setting means depends on the type specified in the metadb_server_cred_type statement. This is ignored when you are using sharedsecret authentication. Please read the description of the spool_server_cred_type statement on the configuration of this statement.
The agent_serverhost statement specifies the host name on which gfarm_agent is running.
For example,
agent_serverhost ldap.example.com
The agent_serverport statement specifies the tcp port number the gfarm_agent is listening on.
For example,
agent_serverport 603
The ldap_serverhost statement specifies the host name on which the LDAP server for filesystem metadata is running.
For example,
ldap_serverhost ldap.example.com
The ldap_serverport statement specifies the tcp port number of the LDAP server.
This statement cannot be omitted if ldap_serverhost is specified.
For example,
ldap_serverport 602
The ldap_base_dn statement specifies the base-distinguished name of the LDAP database.
This statement cannot be omitted if ldap_serverhost is specified.
For example,
ldap_base_dn "dc=example, dc=com"
The ldap_bind_dn statement specifies the distinguished name for the bind operation which is used for authentication to the LDAP database.
For example,
ldap_bind_dn "cn=gfarmuser, dc=example, dc=com"
The ldap_bind_password statement specifies the password for the bind operation which is used for authentication to the LDAP database.
For example,
ldap_bind_password "secret-ldap-password"
The postgresql_serverhost statement specifies the host name on which the PostgreSQL server is running.
For example,
postgresql_serverhost postgresql.example.com
The postgresql_serverport statement specifies the tcp port number of the PostgreSQL server.
This statement cannot be omitted if postgresql_serverhost is specified.
For example,
postgresql_serverport 602
The postgresql_dbname statement specifies the database name of the PostgreSQL database.
This statement cannot be omitted if postgresql_serverhost is specified.
For example,
postgresql_dbname gfarm
The postgresql_user statement specifies the username used to connect the PostgreSQL database.
For example,
postgresql_user gfarm
The postgresql_password statement specifies the password used to connect the PostgreSQL database.
For example,
postgresql_password gfarm
The postgresql_conninfo statement specifies the connection option used to connect the PostgreSQL database.
For example,
postgresql_conninfo "sslmode=require connect_timeout=30"
This statement specifies the authentication method when communicating with the host(s) specified by the third argument.
The first argument should be either the enable or disable keyword. The second argument, auth method, should be the gsi, gsi_auth, or sharedsecret keyword. The third argument specifies the host(s) by using Host specification.
The auth statement may be specified any number of times. For each authentication method, it becomes a candidate when the first entry whose host_specification matches the target host has the enable keyword. When there is no corresponding entry, or when the first corresponding entry has the disable keyword, the authentication method does not become a candidate.
This process takes place on both client and server sides. Candidates for authentication method on both sides will be tried.
The order of statements with different authentication methods is not relevant. When there are several candidates for the authentication method for the host, the order of the authentication trial is sharedsecret, gsi_auth, and then gsi.
The GSI methods are available if and only if the --with-globus option is specified at configuration. When the methods are not available, an auth statement with gsi or gsi_auth will be ignored.
This statement cannot be omitted.
Limitation:
You cannot mix gsi authenticaion and sharedsecret authentication in most cases. You cannot mix gsi_auth and sharedsecret in most cases either.
This is because when a gfarm client like gfrcmd, gfrun, gfrep or gfarmfs with automatic replication enabled (gfarmfs -N) accesses a gfarm filesystem node, the gfarm filesystem node accesses another gfarm filesystem node. If sharedsecret authentication is performed at the access to the first filesystem node, and if gsi (or gsi_auth) authentication is required at the access to the second filesystem node, the access to the second filesystem node will fail, because the gfarm client doesn't pass its GSI proxy certificate to the first gfarm filesystem node (since it doesn't use gsi or gsi_auth but sharedsecret), thus the first filesystem node cannot use any GSI proxy certificate to access the second filesystem node.
That means you can mix sharedsecret and gsi (or gsi_auth) in certain cases, if the second access never will use gsi (or gsi_auth) authentication. For example, if there is only one PC cluster in your system, and if all nodes in the cluster share users' home directories via NFS, and if all gfarm filesystem nodes belong to the cluster, then you can use sharedsecret authentication within the cluster, and gfarm clients at outside of the cluster can access the gfarm filesystem nodes in the cluster by using gsi (or gsi_auth) authentication. The following gfarm.conf setting is an example of such configuration, assuming that the cluster is using 192.168.0.0/24 as its IP address, and requiring gsi authentication from outside of the cluster.
auth enable sharedsecret 192.168.0.0/24
auth enable gsi *
Example:
auth disable sharedsecret 192.168.0.100
auth disable sharedsecret 192.168.0.101
auth enable sharedsecret 192.168.0.0/24
auth enable gsi_auth 10.0.0.0/8
auth enable gsi *
In this example, all hosts which belong to the network address 192.168.0.0/24, except for two hosts, 192.168.0.100 and 192.168.0.101, will be tested for authenticated by both sharedsecret and gsi; all hosts which belong to the network address 10.0.0.0/8 will be tested for authentication by both gsi_auth and gsi; and all other hosts will be authenticated by gsi. Note that two hosts, 192.168.0.100 and 192.168.0.101, will be tested for authentication by gsi only.
Please note that all filesystem nodes must belong to the IP address range 192.168.0.0/24 except 192.168.0.100 and 192.168.0.101 in this example, due to the reason described in the limitation section.
The netparam statement specifies several communication parameters listed below.
parallel_streams. This parameter specifies the number of tcp streams. The default is a single stream. The parallel_streams parameter may improve the file transfer performance, especially when connecting to a distant site (i.e., a high bandwidth-delay product network) such that the tcp congestion window size will not increase sufficiently. This parameter is only effective on a client node where the gfrep(1) command is executed. Host_specification specifies the source host(s) for file replication instead of the destination host(s).
stripe_unit_size. This parameter specifies how to divide the data when utilizing multiple tcp streams. Without this parameter, the data will be divided into parallel_streams of contiguous blocks with almost even size. When this parameter is specified, the data is divided in round-robin fashion with the specified block size in bytes. Generally speaking, the performance of file transfer is improved using this parameter.
This parameter is effective only when two or more is specified as the parallel_streams parameter. This parameter is only effective on a client node where the gfrep(1) command is executed. Host_specification specifies the source host(s) for file replication instead of the destination host(s).
rate_limit. This parameter specifies the maximum transfer rate (bps; bit per second) for a single connection, and is introduced experimentally. The parameter on a source filesystem node is effective during parallel file replication. Host specification specifies the destination host(s) for file replication. Note that this parameter has to be specified on a source filesystem node, not on a client host, unlike the other parameters.
For example,
netparam parallel_streams=2 10.0.0.0/8
netparam stripe_unit_size=8192 10.0.0.0/8
netparam rate_limit=750000 192.168.0.0/24
The sockopt parameter specifies the socket option option via the setsockopt(2) system call.
When LISTENER (all capital letters) is specified by the second argument, the socket option is applied to any socket on the server side (accepting side).
When the host_specification is specified by the second argument, the socket option is applied to sockets that connect to the specified host(s). If the second argument is "*", the socket option is applied to any hosts on the client side (connecting side).
If the second argument is omitted, the socket option is applied to every socket.
The following socket options can be specified.
debug. The SO_DEBUG socket
option is specified. A value is not
necessary.
keepalive. The SO_KEEPALIVE
socket option is specified. A value is not
necessary.
sndbuf. The SO_SNDBUF socket
option is specified with a value.
rcvbuf. The SO_RCVBUF socket
option is specified with a value.
tcp_nodelay. The TCP_NODELAY
socket option is specified. A value is not
necessary.
For example,
sockopt tcp_nodelay 192.168.0.0/24
sockopt sndbuf=1048576 10.0.0.0/8
sockopt sndbuf=1048576 LISTENER
sockopt rcvbuf=1048576 10.0.0.0/8
sockopt rcvbuf=1048576 LISTENER
The address_use statement specifies an IP address or a network address that is given preference for connection when the target host has several IP addresses.
For example,
address_use 192.168.0.0/24
This directive specifies the default architecture name that may be used when a client node accesses an executable file. This directive only affects client nodes. For filesystem nodes, architecture names registered in the metadata server are used.
On the other hand, the environment variable, GFARM_ARCHITECTURE, which specifies the default architecture name, just as in this directive, affects both client nodes and filesystem nodes.
Example:
client_architecture i386-redhat8.0-linux linux-1.example.com
client_architecture i386-redhat8.0-linux linux-2.example.com
client_architecture sparc-sun-solaris8 solaris-1.example.com
This directive specifies a file name user-map-file for mapping global user names to local user names
This directive is optional. user-map-file is needed when you have to use the sharedsecret authentication method in the case where you have different unix account names on different filesystem nodes. In such a case, the user-map-file on each filesystem node should have an entry from a unique global user name to each local user name.
Example:
local_user_map /etc/gfarm/gfarm-usermap
Each line of the user-map-file consists of two fields separated by spaces; the first field is a global user name, and the second field is a local user name.
Example of the user mapping file:
foobar foo
quux baz
According to the first line of this mapping file, a global user name, "foobar", is mapped to a local user name, "foo", on this node.
This directive specifies the time (in seconds) until a directory cache in the gfarm library expires. The default time is 86,400 seconds, i.e. one day. The time can be specified by an environment variable, GFARM_DIRCACHE_TIMEOUT, too. The environment variable takes precedence over the directive.
For example,
dir_cache_timeout 60
This directive specifies the time (in seconds) until the metadata cache of the filesystem nodes in the gfarm library expires. Newly added or removed hosts might not be seen until this timeout has passed. The default time is 600 seconds, i.e. ten minutes.
For example,
host_cache_timeout 60
This directive specifies the time (in seconds) until the cache used for filesystem node scheduling expires. The cache holds information on each filesystem node, e.g. load average, disk free space, and whether authentication succeeds or not. The default time is 600 seconds, i.e. ten minutes.
For example,
schedule_cache_timeout 60
Gfarm nearly always chooses localhost, when it selects a filesystem node to write a file. By specifing the disable keyword in this directive, Gfarm handles localhost the same as other filesystem nodes. The same thing can be achieved by specifying disable as the value of the GFARM_WRITE_LOCAL_PRIORITY environment variable. When both this directive and the environment variable are specified, the value specified by the environment variable will be used. The default is enable.
For example,
write_local_priority disable
Gfarm tries to choose a filesystem node in the domain specified by this directive, when it selects a node to write a file. If there is no suitable node for writing in this domain, or if this directive is not specified, the filesystem node will be chosen from among all nodes. You can specify this domain with the GFARM_WRITE_TARGET_DOMAIN environment variable too. When both this directive and the environment variable are specified, the value specified by the environment variable will be used. Note that if you specify this directive more than once, only the first one will be used and the others are ignored.
For example,
write_target_domain example.org
This directive specifies free disk space (in bytes) which is required on filesystem nodes. The Gfarm scheduler excludes filesystem nodes which have less free space than this parameter, when it schedules nodes for jobs which may write files. The free space value may have a suffix like ``k'' (kilo bytes), ``M'' (mega bytes), ``G'' (giga bytes) and ``T'' (tera bytes). The default size is 128M bytes.
For example,
minimum_free_disk_space 1G
This directive specifies maximum number of cached gfsd connections. The default is 16.
For example,
gfsd_connection_cache 32
This directive specifies a level of log priority. The log output, which priority is inferior to this level, will not be sent to syslog or standard error. The priority levels are "emerg", "alert", "crit", "err", "warning", "notice", "info" and "debug" in highest first order. The default level is "info". It's not recommended to specify a level higher or equal to "crit".
For example,
log_level debug
When "enabled" is specified, Gfarm records last access time strictly. When "disabled" is specified, the access time becomes inaccurate, but such setting reduces the load of metadata updates, and may improve performance. The "disable" setting corresponds to "noatime" setting in usual filesystem.
Example:
record_atime disable
This is a grammar for gfarm.conf described using BNF notation.
<statement> ::=
<spool_statement> |
<spool_server_listen_address_statement> |
<spool_serverport_statement> |
<spool_server_cred_type_statement> |
<spool_server_cred_service_statement> |
<spool_server_cred_name_statement> |
<metadb_serverhost_statement> |
<metadb_serverport_statement> |
<metadb_server_cred_type_statement> |
<metadb_server_cred_service_statement> |
<metadb_server_cred_name_statement> |
<agent_serverhost_statement> |
<agent_serverport_statement> |
<ldap_serverhost_statement> |
<ldap_serverport_statement> |
<ldap_base_dn_statement> |
<ldap_bind_dn_statement> |
<ldap_bind_password_statement> |
<postgresql_serverhost_statement> |
<postgresql_serverport_statement> |
<postgresql_dbname_statement> |
<postgresql_user_statement> |
<postgresql_password_statement> |
<postgresql_conninfo_statement> |
<auth_statement> |
<netparam_statement> |
<sockopt_statement> |
<address_use_statement> |
<client_architecture_statement> |
<option_statement> |
<local_user_map_statement>
<dir_cache_timeout_statement> |
<host_cache_timeout_statement> |
<schedule_cache_timeout_statement> |
<write_local_priority_statement> |
<write_target_domain_statement> |
<minimum_free_disk_space_statement> |
<gfsd_connection_cache_statement> |
<log_level_statement> |
<record_atime_statement>
<spool_statement> ::= "spool" <pathname>
<spool_server_listen_address_statement> ::=
"spool_server_listen_address" <ipv4_address>
<spool_serverport_statement> ::= "spool_serverport" <hostname>
<spool_server_cred_type_statement> ::=
"spool_server_cred_type" <cred_type>
<spool_server_cred_service_statement> ::=
"spool_server_cred_service" <cred_service>
<spool_server_cred_name_statement> ::=
"spool_server_cred_name" <cred_name>
<metadb_serverhost_statement> ::= "metadb_serverhost" <hostname>
<metadb_serverport_statement> ::= "metadb_serverport" <portnumber>
<metadb_server_cred_type_statement> ::=
"metadb_server_cred_type" <cred_type>
<metadb_server_cred_service_statement> ::=
"metadb_server_cred_service" <cred_service>
<metadb_server_cred_name_statement> ::=
"metadb_server_cred_name" <cred_name>
<agent_serverhost_statement> ::= "agent_serverhost" <hostname>
<agent_serverport_statement> ::= "agent_serverport" <portnumber>
<ldap_serverhost_statement> ::= "ldap_serverhost" <hostname>
<ldap_serverport_statement> ::= "ldap_serverport" <portnumber>
<ldap_base_dn_statement> ::= "ldap_base_dn" <string>
<ldap_bind_dn_statement> ::= "ldap_bind_dn" <string>
<ldap_bind_password_statement> ::= "ldap_bind_password" <string>
<postgresql_serverhost_statement> ::= "postgresql_serverhost" <hostname>
<postgresql_serverport_statement> ::= "postgresql_serverport" <portnumber>
<postgresql_dbname_statement> ::= "postgresql_dbname" <string>
<postgresql_user_statement> ::= "postgresql_user" <string>
<postgresql_password_statement> ::= "postgresql_password" <string>
<postgresql_conninfo_statement> ::= "postgresql_conninfo" <string>
<auth_statement> ::=
"auth" <validity> <auth_method> <hostspec>
<auth_method> ::= "gsi" | "gsi_auth" | "sharedsecret"
<netparam_statement> ::=
"netparam" <netparam_option>=<number> [<hostspec>]
<netparam_option> ::= "parallel_streams" | "stripe_unit_size" |
"rate_limit"
<sockopt_statement> ::=
"sockopt" <socket_option>[=<number>] [""LISTENER" | <hostspec>]
<socket_option> = "debug" | "keepalive" | "sndbuf" | "rcvbuf" |
"tcp_nodelay"
<address_use_statement> ::= "address_use" <hostspec>
<client_architecture_statement> ::=
"client_architecture" <architecture> <hostspec>
<option_statement> ::= "option" <an_option>
<an_option> ::= "noatime"
<local_user_map_statement> ::= "local_user_map" <pathname>
<dir_cache_timeout_statement> ::= "dir_cache_timeout" <number>
<host_cache_timeout_statement> ::= "host_cache_timeout" <number>
<schedule_cache_timeout_statement> ::= "schedule_cache_timeout" <number>
<write_local_priority_statement> ::= "write_local_priority" <validity>
<write_target_domain_statement> ::= "write_target_domain" <string>
<minimum_free_disk_space_statement> ::=
"minimum_free_disk_space" <size>
<gfsd_connection_cache_statement> ::= "gfsd_connection_cache" <number>
<log_level_statement> ::= "log_level" <log_priority>
<record_atime_statement> ::= "record_atime" <validity>
<hostspec> ::= <ipv4_address> | <ipv4_address> "/" <address_mask> |
<hostname> | "." <domain_name> | "*"
<pathname> ::= <pathname_character> <pathname_character>*
<pathname_character> ::= <hostname_character> | "," | "/" | "_"
<hostname> ::= <hostname_character> <hostname_character>*
<hostname_character> ::= <alphabet> | <digit> | "-" | "."
<portnumber> ::= <number>
<size> ::= <number> [ "k" | "M" | "G" | "T" ]
<number> ::= <digit> [<digit>*]
<digit> ::= "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"
<string> ::= """ <double_quoted_character>* """
<double_quoted_character> ::=
<any_character_except_backslash_and_double_quotation> |
"\\" | "\""
<validity> ::= "enable" | "disable"
<log_priority> ::= "emerg" | "alert" | "crit" | "err" | "warning" |
"notice" | "info" | "debug"
The following is an example usin PostgreSQL to store the metadata, and to allow access from filesystem nodes and clients at IP address 192.168.0.0/24, via sharedsecret authentication.
spool /var/spool/gfarm
spool_serverport 600
metadb_serverhost metadb.example.org
metadb_serverport 601
postgresql_serverhost metadb.example.org
postgresql_serverport 5432
postgresql_dbname gfarm
postgresql_user gfarm
postgresql_password "secret-postgresql-password"
auth enable sharedsecret 192.168.0.0/24
sockopt keepalive
The following is an example using LDAP to store the metadata, and to allow access from filesystem nodes and clients at any IP address, via GSI authentication.
spool /var/spool/gfarm
spool_serverport 600
metadb_serverhost metadb.example.com
metadb_serverport 601
ldap_serverhost metadb.example.com
ldap_serverport 389
ldap_base_dn "dc=example, dc=com"
ldap_bind_dn "cn=gfarmuser, dc=example, dc=com"
ldap_bind_password "secret-ldap-password"
auth enable gsi *
sockopt keepalive