Nothing excites me than being able to solve a problem on Linux just before bed time. Well, there could be other things, but I couldn’t think of any right now.

Right now, I’m preparing for my next exam - Red Hat Certified Specialist in Identity Management (EX362), and one of the topics is about setting up automated home directory using IdM (or IPA). I tried to follow the course material, but I had troubles getting it working.

In this post, I’m going to describe how to do this, the issues I came across and the solution. In fact, I found the solution from another blog post. This really makes me happy that someone’s blog post fixes my headache. So who knows, maybe this blog post will help saving you from losing some hair too.

The assumption:

  • The IdM/IPA server is idm.lab.example.net
  • The domain is: lab.example.net
  • The Kerberos Realm is LAB.EXAMPLE.NET
  • The exported home directory is on /exports/home
  • The client is client.lab.example.net
  • The NFS server is utility.lab.example.net

Part 1 - Configure Kerberized NFS Server

[root@utility ~]# kinit admin
Password for admin@LAB.EXAMPLE.NET: 
[root@utility ~]# klist
Ticket cache: KEYRING:persistent:0:0
Default principal: admin@LAB.EXAMPLE.NET

Valid starting       Expires              Service principal
09/24/2020 12:19:35  09/25/2020 12:19:30  krbtgt/LAB.EXAMPLE.NET@LAB.EXAMPLE.NET
[root@utility ~]# 

Create a service principal for the NFS service on utility.lab.example.net.

[root@utility ~]# ipa service-add nfs/utility.lab.example.net
-----------------------------------------------------------
Added service "nfs/utility.lab.example.net@LAB.EXAMPLE.NET"
-----------------------------------------------------------
  Principal name: nfs/utility.lab.example.net@LAB.EXAMPLE.NET
  Principal alias: nfs/utility.lab.example.net@LAB.EXAMPLE.NET
  Managed by: utility.lab.example.net
[root@utility ~]# 

Generate, and retrieve the keytab for the NFS service principal.

[root@utility ~]# ipa-getkeytab -s idm.lab.example.net \
> -p nfs/utility.lab.example.net \
> -k /etc/krb5.keytab 
Keytab successfully retrieved and stored in: /etc/krb5.keytab

Setup the NFS server on utility and configure the firewall rules.

[root@utility ~]# yum list nfs-utils
Installed Packages
nfs-utils.x86_64                                                    1:1.3.0-0.66.el7                                                     @base
[root@utility ~]# 

Configure firewalld on the NFS server. There are 3 services we need to open up on the firewall: nfs, mountd, and rpc-bind.

[root@utility ~]# firewall-cmd --add-service=nfs \
--add-service=mountd --add-service=rpc-bind
[root@utility ~]# !! --permanent

On utility, as the root user, enable secure NFS and set the IPA DNS domain.

[root@utility ~]# ipa-client-automount 
Searching for IPA server...
IPA server: DNS discovery
Location: default
Continue to configure the system with these values? [no]: yes
Configured /etc/sysconfig/nfs
Configured /etc/idmapd.conf
Started rpcidmapd
Started rpcgssd
Restarting sssd, waiting for it to become available.
Started autofs

On utility, configure a Kerberos-aware NFS export of the /export/home directory to be used.

[root@utility ~]# mkdir -p /export/home

[root@utility ~]# echo "/export/home *(rw,sec=krb5:krb5i:krb5p)" >> /etc/exports
/export/home *(rw,sec=krb5:krb5i:krb5p)

Export the new share using exportfs command.

[root@utility ~]# exportfs -ra

Enable and start the NFS server and related services.

[root@utility ~]# systemctl start nfs-server
[root@utility ~]# systemctl start nfs-secure
[root@utility ~]# showmount -e
Export list for utility.lab.example.net:
/export/home *

On utility, create the automount map and key. Create the auto.home automount map in the default location.

[root@utility ~]# ipa automountmap-add-indirect default auto.home --mount=/home
-------------------------------
Added automount map "auto.home"
-------------------------------
  Map: auto.home

Create automount key.

[root@utility ~]# ipa automountkey-add default auto.home --key "*" \
> --info "utility.lab.example.net:/export/home/&"
-----------------------
Added automount key "*"
-----------------------
  Key: *
  Mount information: utility.lab.example.net:/export/home/&

Part 2 - Create a user and its home directory

We can create a user called nfsuser01 from any server that we can get the admin kerberos certificate. In my example, I just create the user on the utility server.

[root@utility ~]# ipa user-add --first nfsuser01 --last nfsuser --password CentOs123^

Lastly, let’s create a “roaming” home directory for the nfsuser01 user. We’ll do this on utility, since it’s where the NFS share resides on.

[root@utility ~]# mkhomedir_helper nfsuser01
[root@utility ~]# mv /home/nfsuser01 /export/home
[root@utility ~]# chmod 0700 /export/home/nfsuser01
[root@utility ~]# echo "Hello there" > /export/home/nfsuser01/README.txt
[root@utility ~]# chown nfsuser01: /export/home/nfsuser01/README.txt

Part 3 - Configure a Kerberized NFS client

On the client, configure its kerberized NFS settings by running ipa-client-automount.

[root@client ~]# ipa-client-automount
Searching for IPA server...
IPA server: DNS discovery
Location: default
Continue to configure the system with these values? [no]: yes
Configured /etc/sysconfig/nfs
Configured /etc/idmapd.conf
Started rpcidmapd
Started rpcgssd
Restarting sssd, waiting for it to become available.
Started autofs

Enable and start necessary services. According to the Red Hat learning material, 2 services are provided by nfs-utils package (rpc-gssd and nfs-idmapd).

[root@client ~]# rpm -ql nfs-utils | grep service
/usr/lib/systemd/system/auth-rpcgss-module.service
/usr/lib/systemd/system/nfs-blkmap.service
/usr/lib/systemd/system/nfs-config.service
/usr/lib/systemd/system/nfs-idmap.service
/usr/lib/systemd/system/nfs-idmapd.service
/usr/lib/systemd/system/nfs-lock.service
/usr/lib/systemd/system/nfs-mountd.service
/usr/lib/systemd/system/nfs-secure.service
/usr/lib/systemd/system/nfs-server.service
/usr/lib/systemd/system/nfs-utils.service
/usr/lib/systemd/system/nfs.service
/usr/lib/systemd/system/nfslock.service
/usr/lib/systemd/system/rpc-gssd.service
/usr/lib/systemd/system/rpc-statd-notify.service
/usr/lib/systemd/system/rpc-statd.service
/usr/lib/systemd/system/rpcgssd.service
/usr/lib/systemd/system/rpcidmapd.service

Another service is provided by rpcbind package.

[root@client ~]# rpm -ql rpcbind | grep service
/usr/lib/systemd/system/rpcbind.service

Let’s enable and start them in one go.

[root@client ~]# systemctl enable rpc-gssd nfs-idmapd rpcbind --now
Created symlink from /etc/systemd/system/multi-user.target.wants/rpcbind.service to /usr/lib/systemd/system/rpcbind.service.
[root@client ~]# systemctl status rpc-gssd
● rpc-gssd.service - RPC security service for NFS client and server
   Loaded: loaded (/usr/lib/systemd/system/rpc-gssd.service; static; vendor preset: disabled)
   Active: active (running) since Thu 2020-09-24 13:46:00 AEST; 6min ago
 Main PID: 708 (rpc.gssd)
   CGroup: /system.slice/rpc-gssd.service
           └─708 /usr/sbin/rpc.gssd

Sep 24 13:46:00 client.lab.example.net systemd[1]: Starting RPC security service for NFS client and server...
Sep 24 13:46:00 client.lab.example.net systemd[1]: Started RPC security service for NFS client and server.
[root@client ~]# systemctl status rpcbind
● rpcbind.service - RPC bind service
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-09-24 13:48:19 AEST; 4min 44s ago
 Main PID: 1180 (rpcbind)
   CGroup: /system.slice/rpcbind.service
           └─1180 /sbin/rpcbind -w

Sep 24 13:48:18 client.lab.example.net systemd[1]: Starting RPC bind service...
Sep 24 13:48:19 client.lab.example.net systemd[1]: Started RPC bind service.
[root@client ~]# systemctl status nfs-idmapd
● nfs-idmapd.service - NFSv4 ID-name mapping service
   Loaded: loaded (/usr/lib/systemd/system/nfs-idmapd.service; static; vendor preset: disabled)
   Active: active (running) since Thu 2020-09-24 13:52:26 AEST; 46s ago
  Process: 1243 ExecStart=/usr/sbin/rpc.idmapd $RPCIDMAPDARGS (code=exited, status=0/SUCCESS)
 Main PID: 1247 (rpc.idmapd)
   CGroup: /system.slice/nfs-idmapd.service
           └─1247 /usr/sbin/rpc.idmapd

Sep 24 13:52:26 client.lab.example.net systemd[1]: Starting NFSv4 ID-name mapping service...
Sep 24 13:52:26 client.lab.example.net systemd[1]: Started NFSv4 ID-name mapping service.

Note: The states for rpc-gssd.service and nfs-idmapd.service are “static”. Usually, we can’t enable or disable a static service. What’s going on here?

Part 4 - Test and verify that everything works as expected

[student@workstation ~]$ ssh -o PubkeyAuthentication=no nfsuser01@client.lab.example.net
Password: 
Last login: Thu Sep 24 01:04:09 2020 from workstation.lab.example.com
-sh-4.2$ ls

It is not working!!!!

-sh-4.2$ mount | grep nfs
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
-sh-4.2$ pwd
/home/nfsuser01
-sh-4.2$ 

To be honest, I was stuck here for an hour or so. I tried many things from uninstalling the ipa-client-automount, restart services..etc. Then I found this blog post Using IPA to provide automount maps for NFSv4 home directories from 2015 which pointed me to the right direction.

According to that blog post, the ipa-client-automount didn’t update the /etc/nswitch.conf correctly.

[root@client ~]# grep automount /etc/nsswitch.conf
automount:  files

It should have been:

automount:  files sss

So let’s fix this, and restart autofs.service to pick up this change.

[root@client ~]# sed -i 's/automount:  files/automount:  files sss/' /etc/nsswitch.conf
[root@client ~]# grep automount /etc/nsswitch.conf -n
62:automount:  files sss
[root@client ~]# systemctl restart autofs

Here is the result after login to client using nfsuser01.

[student@workstation ~]$ ssh -o PubkeyAuthentication=no nfsuser01@client.lab.example.net
Password: 
Last login: Thu Sep 24 14:38:30 2020 from workstation.lab.example.com
-sh-4.2$ ls
README.txt
-sh-4.2$ cat README.txt 
Hello there
-sh-4.2$ mount | grep nfs
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
utility.lab.example.net:/export/home/nfsuser01 on /home/nfsuser01 type nfs4 (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,clientaddr=172.25.250.11,local_lock=none,addr=172.25.250.14)
-sh-4.2$

As mentioned in the beginning, I’m quite happy that I was able to get this working. But what’s made me even happier is that someone’s blog post from 2015 could give me hint to fix the issue in 2020. Well, this is one of the reasons why I keep this blog running too. :)

References