Nothing excites me than being able to solve a problem on Linux just before bed time. Well, there could be other things, but I couldn’t think of any right now.
Right now, I’m preparing for my next exam - Red Hat Certified Specialist in Identity Management (EX362), and one of the topics is about setting up automated home directory using IdM (or IPA). I tried to follow the course material, but I had troubles getting it working.
In this post, I’m going to describe how to do this, the issues I came across and the solution. In fact, I found the solution from another blog post. This really makes me happy that someone’s blog post fixes my headache. So who knows, maybe this blog post will help saving you from losing some hair too.
The assumption:
- The IdM/IPA server is
idm.lab.example.net - The domain is:
lab.example.net - The Kerberos Realm is
LAB.EXAMPLE.NET - The exported home directory is on
/exports/home - The client is
client.lab.example.net - The NFS server is
utility.lab.example.net
Part 1 - Configure Kerberized NFS Server
[root@utility ~]# kinit admin
Password for admin@LAB.EXAMPLE.NET:
[root@utility ~]# klist
Ticket cache: KEYRING:persistent:0:0
Default principal: admin@LAB.EXAMPLE.NET
Valid starting Expires Service principal
09/24/2020 12:19:35 09/25/2020 12:19:30 krbtgt/LAB.EXAMPLE.NET@LAB.EXAMPLE.NET
[root@utility ~]#
Create a service principal for the NFS service on utility.lab.example.net.
[root@utility ~]# ipa service-add nfs/utility.lab.example.net
-----------------------------------------------------------
Added service "nfs/utility.lab.example.net@LAB.EXAMPLE.NET"
-----------------------------------------------------------
Principal name: nfs/utility.lab.example.net@LAB.EXAMPLE.NET
Principal alias: nfs/utility.lab.example.net@LAB.EXAMPLE.NET
Managed by: utility.lab.example.net
[root@utility ~]#
Generate, and retrieve the keytab for the NFS service principal.
[root@utility ~]# ipa-getkeytab -s idm.lab.example.net \
> -p nfs/utility.lab.example.net \
> -k /etc/krb5.keytab
Keytab successfully retrieved and stored in: /etc/krb5.keytab
Setup the NFS server on utility and configure the firewall rules.
[root@utility ~]# yum list nfs-utils
Installed Packages
nfs-utils.x86_64 1:1.3.0-0.66.el7 @base
[root@utility ~]#
Configure firewalld on the NFS server. There are 3 services we need to open up on the firewall: nfs, mountd, and rpc-bind.
[root@utility ~]# firewall-cmd \
--add-service=nfs --add-service=mountd --add-service=rpc-bind
[root@utility ~]# !! --permanent
(Note: !! is a bash-shortcut for repeating the previous running command. What I wanted is to run previous command and append –permanent option to it.)
On utility, run ipa-client-automount to enable secure NFS and set the IdM/IPA DNS domain.
[root@utility ~]# ipa-client-automount
Searching for IPA server...
IPA server: DNS discovery
Location: default
Continue to configure the system with these values? [no]: yes
Configured /etc/sysconfig/nfs
Configured /etc/idmapd.conf
Started rpcidmapd
Started rpcgssd
Restarting sssd, waiting for it to become available.
Started autofs
On utility, configure a Kerberos-aware NFS export of the /export/home directory to be used.
[root@utility ~]# mkdir -p /export/home
[root@utility ~]# echo "/export/home *(rw,sec=krb5:krb5i:krb5p)" >> /etc/exports
[root@utility ~]# cat /etc/exports
/export/home *(rw,sec=krb5:krb5i:krb5p)
Export the new share using exportfs command. (-r: Reexport all directories, -a: Export or unexport all directories.)
[root@utility ~]# exportfs -ra
Enable and start the NFS server and related services.
[root@utility ~]# systemctl enable nfs-server --now
[root@utility ~]# systemctl enable nfs-secure --now
[root@utility ~]# showmount -e
Export list for utility.lab.example.net:
/export/home *
On utility, create the automount map and key. First, create the auto.home automount map in the default location.
[root@utility ~]# ipa automountmap-add-indirect default auto.home --mount=/home
-------------------------------
Added automount map "auto.home"
-------------------------------
Map: auto.home
Next, create an automount key.
[root@utility ~]# ipa automountkey-add default auto.home --key "*" \
> --info "utility.lab.example.net:/export/home/&"
-----------------------
Added automount key "*"
-----------------------
Key: *
Mount information: utility.lab.example.net:/export/home/&
Part 2 - Create a user and its home directory
We can create a user called nfsuser01 from any server that we can get the admin kerberos credential. In my example, a new test user was created on the utility server.
[root@utility ~]# ipa user-add --first nfsuser01 --last nfsuser --password CentOs123^
Lastly, let’s create a “roaming” home directory for the nfsuser01 user. We’ll do this on utility, since it’s where the NFS share resides on.
[root@utility ~]# mkhomedir_helper nfsuser01
[root@utility ~]# mv /home/nfsuser01 /export/home
[root@utility ~]# chmod 0700 /export/home/nfsuser01
[root@utility ~]# echo "Hello there" > /export/home/nfsuser01/README.txt
[root@utility ~]# chown nfsuser01: /export/home/nfsuser01/README.txt
Part 3 - Configure a Kerberized NFS client
On the client, configure its kerberized NFS settings by running ipa-client-automount.
[root@client ~]# ipa-client-automount
Searching for IPA server...
IPA server: DNS discovery
Location: default
Continue to configure the system with these values? [no]: yes
Configured /etc/sysconfig/nfs
Configured /etc/idmapd.conf
Started rpcidmapd
Started rpcgssd
Restarting sssd, waiting for it to become available.
Started autofs
Enable and start necessary services. According to the Red Hat learning material, 2 services are provided by nfs-utils package (rpc-gssd and nfs-idmapd).
[root@client ~]# rpm -ql nfs-utils | grep service
/usr/lib/systemd/system/auth-rpcgss-module.service
/usr/lib/systemd/system/nfs-blkmap.service
/usr/lib/systemd/system/nfs-config.service
/usr/lib/systemd/system/nfs-idmap.service
/usr/lib/systemd/system/nfs-idmapd.service
/usr/lib/systemd/system/nfs-lock.service
/usr/lib/systemd/system/nfs-mountd.service
/usr/lib/systemd/system/nfs-secure.service
/usr/lib/systemd/system/nfs-server.service
/usr/lib/systemd/system/nfs-utils.service
/usr/lib/systemd/system/nfs.service
/usr/lib/systemd/system/nfslock.service
/usr/lib/systemd/system/rpc-gssd.service
/usr/lib/systemd/system/rpc-statd-notify.service
/usr/lib/systemd/system/rpc-statd.service
/usr/lib/systemd/system/rpcgssd.service
/usr/lib/systemd/system/rpcidmapd.service
Another service is provided by rpcbind package.
[root@client ~]# rpm -ql rpcbind | grep service
/usr/lib/systemd/system/rpcbind.service
Let’s enable and start them in one go.
[root@client ~]# systemctl enable rpc-gssd nfs-idmapd rpcbind --now
Created symlink from /etc/systemd/system/multi-user.target.wants/rpcbind.service to /usr/lib/systemd/system/rpcbind.service.
[root@client ~]# systemctl status rpc-gssd
● rpc-gssd.service - RPC security service for NFS client and server
Loaded: loaded (/usr/lib/systemd/system/rpc-gssd.service; static; vendor preset: disabled)
Active: active (running) since Thu 2020-09-24 13:46:00 AEST; 6min ago
Main PID: 708 (rpc.gssd)
CGroup: /system.slice/rpc-gssd.service
└─708 /usr/sbin/rpc.gssd
Sep 24 13:46:00 client.lab.example.net systemd[1]: Starting RPC security service for NFS client and server...
Sep 24 13:46:00 client.lab.example.net systemd[1]: Started RPC security service for NFS client and server.
[root@client ~]# systemctl status rpcbind
● rpcbind.service - RPC bind service
Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-09-24 13:48:19 AEST; 4min 44s ago
Main PID: 1180 (rpcbind)
CGroup: /system.slice/rpcbind.service
└─1180 /sbin/rpcbind -w
Sep 24 13:48:18 client.lab.example.net systemd[1]: Starting RPC bind service...
Sep 24 13:48:19 client.lab.example.net systemd[1]: Started RPC bind service.
[root@client ~]# systemctl status nfs-idmapd
● nfs-idmapd.service - NFSv4 ID-name mapping service
Loaded: loaded (/usr/lib/systemd/system/nfs-idmapd.service; static; vendor preset: disabled)
Active: active (running) since Thu 2020-09-24 13:52:26 AEST; 46s ago
Process: 1243 ExecStart=/usr/sbin/rpc.idmapd $RPCIDMAPDARGS (code=exited, status=0/SUCCESS)
Main PID: 1247 (rpc.idmapd)
CGroup: /system.slice/nfs-idmapd.service
└─1247 /usr/sbin/rpc.idmapd
Sep 24 13:52:26 client.lab.example.net systemd[1]: Starting NFSv4 ID-name mapping service...
Sep 24 13:52:26 client.lab.example.net systemd[1]: Started NFSv4 ID-name mapping service.
Note: The states for rpc-gssd.service and nfs-idmapd.service are “static”. Usually, we can’t enable or disable a static service. What’s going on here?
Part 4 - Test and verify that everything works as expected
[student@workstation ~]$ ssh -o PubkeyAuthentication=no nfsuser01@client.lab.example.net
Password:
Last login: Thu Sep 24 01:04:09 2020 from workstation.lab.example.com
-sh-4.2$ ls
It is not working!!!!
-sh-4.2$ mount | grep nfs
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
-sh-4.2$ pwd
/home/nfsuser01
-sh-4.2$
To be honest, I was stuck here for an hour or so. I tried many things from uninstalling the ipa-client-automount, restart services..etc. Then I found this blog post Using IPA to provide automount maps for NFSv4 home directories from 2015 which pointed me to the right direction.
According to that blog post, the ipa-client-automount didn’t update the /etc/nswitch.conf correctly.
[root@client ~]# grep automount /etc/nsswitch.conf
automount: files
It should have been:
automount: files sss
So let’s fix this, and restart autofs.service to pick up this change.
[root@client ~]# sed -i 's/automount: files/automount: files sss/' /etc/nsswitch.conf
[root@client ~]# grep automount /etc/nsswitch.conf -n
62:automount: files sss
[root@client ~]# systemctl restart autofs
Here is the result after login to client using nfsuser01.
[student@workstation ~]$ ssh -o PubkeyAuthentication=no nfsuser01@client.lab.example.net
Password:
Last login: Thu Sep 24 14:38:30 2020 from workstation.lab.example.com
-sh-4.2$ ls
README.txt
-sh-4.2$ cat README.txt
Hello there
-sh-4.2$ mount | grep nfs
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
utility.lab.example.net:/export/home/nfsuser01 on /home/nfsuser01 type nfs4 (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,clientaddr=172.25.250.11,local_lock=none,addr=172.25.250.14)
-sh-4.2$
As mentioned in the beginning, I’m quite happy that I was able to get this working. But what’s made me even happier is that someone’s blog post from 2015 could give me hint to fix the issue in 2020. Well, this is one of the reasons why I keep this blog running too. :)