At my current work place, we use Puppet to manage desktop machines. Recently, I need to prepare an new image based on Ubuntu 14.04 LTS.
To prepare an image, in brief, first thing is to install a fresh Ubuntu on a machine, compile and package lots of applications, and make change to configuration such as look and feels, etc. As mentioned earlier, I use Puppet to manage machine, so I store all customed configuration on the Puppet server.
While working on this image, I came across a problem with Puppet agent, the software running on each desktop. The issue is that it took very long to execute the code. Here is an example:
matht336:~# puppet agent -vt -l console
Info: Loading facts in /var/lib/puppet/lib/facter/sku.rb
Info: Loading facts in /var/lib/puppet/lib/facter/users.rb
Info: Loading facts in /var/lib/puppet/lib/facter/zid.rb
Info: Loading facts in /var/lib/puppet/lib/facter/videocard.rb
Info: Caching catalog for matht336
Info: Applying configuration version '1398116230'
Notice: Finished catalog run in 1.09 seconds
According to the above output, it only took 1.09 seconds to run the catalog. Well the truth is it took way longer than that, approximately about 20 minutes. I re-ran puppet agent with -d flag for debug, but none of the output information gave me any clues of what could possibily go wrong.
One clue that I’ve been missing and should have noticed much earlier is that puppet agent got stuck after loading the facts.
So I modified /etc/puppet/puppet.conf on the node (not the server), to not receiving custom facts from the server:
[main]
...
pluginsync=false
Then, I removed the custom facts one by one from /var/lib/puppet/facter and re-run puppet agent on the node. Suprisingly, zid.rb was the source of the puppet agent slowness. zid.rb is a fact written by my colleague to keep track of who last logged in to a machine.
I’ve been wondering why noone on the Internet has the same issue as me. At first I thought maybe Ubuntu 14.04 is too and not many people have been running puppet on it. Now I know, it the problem is unique to my environment.
So if you experience the same problem as me, I would suggest that the first thing to check out is openning up thos custom facts and see what they’re doing.