Tag Archives: puppet

Puppetserver failed to start after upgrading to 2.5.0

I upraded puppetserver from 2.4.0 to 2.5.0 on a server running Centos 6.8. Then, puppetserver no longer starts.

The log (I think it was puppetsrver-daemon.log) said:

Exception in thread "main" java.lang.IllegalArgumentException: Specified bootstrap config file does not exist: '/etc/puppetlabs/puppetserver/bootstrap.cfg'

As the error reported, /etc/puppetlabs/puppetserver/bootstrap.cfg was not there. The disappearance must have happened during the puppetserver upgrade. I can confirm this by looking at my backup directory and finding the boostrap.cfg file there: /srv/backup/etc/puppetlabs/puppetserver/bootstrap.cfg.

According to this issue #1247, the boostrap.cfg file is broken into 2 files and relocated to a different directories. This issue was raised sometimes in May, and the problem only happened on my server today in August. How is that possible? This should affect more people right? So far, I haven’t seen any one reported it yet.

I need to get puppetserver running again and fast. Here is a quick fix.

Edit /etc/sysconfig/puppetserver, and make change to BOOTSTRAP_CONFIG variable:

#BOOTSTRAP_CONFIG="/etc/puppetlabs/puppetserver/bootstrap.cfg"
BOOTSTRAP_CONFIG="/etc/puppetlabs/puppetserver/services.d/,${INSTALL_DIR}/config/services.d/"

Start puppetserver and it is working again (for now).

# service puppetserver start
Starting puppetserver:                                     [  OK  ]

Puppet service resource provider

Well, first of all I think the title of this post won’t entirely reflect its content. However, I need to go ahead and post this now.

At work, We’ve been having issues with cups service not running on some Kubuntu 14.04 desktops. This should not happen since even if cups service is dead, we have Puppet agent running on each machine
to check and make sure that ‘cups’ is running else will be automatically restarted.

Here’s the snippet of cups::service manifest:

#-
class cups::service {
  
  service { 'cups':
    ensure     => running,
    enable     => true,
    hasstatus  => true,
    hasrestart => true,
  }

}

I still don’t know what cause cups to die or terminated. However, when that happens, I manually check
cups service by running:

# service cups status
cups start/running,

According to the output, it looks like cups is still running. But if instead I run:

# /etc/init.d/cups status
 * cupsd is not running

To fix this cups issue, cups service needs to be started:

# /etc/init.d/cups start
# /etc/init.d/cups status
 * cups is running

It seems the upstart service used on Kubuntu to check status for cups is broken. Surprisingly, I haven’t found
any bugs report about this yet. Maybe this problem is unique to my environment; or maybe I should file a bug myself!

Anyway, a work around is force Puppet to start cups service with init.d instead of using upstart. In Puppet, we can do this by setting the provider attribute of service resource.

#-
class cups::service {
  
  service { 'cups':
    ensure     => running,
    enable     => true,
    hasstatus  => true,
    hasrestart => true,
    provider   => init,
  }

}

The above Puppet code works fine with Puppet agent 3.8.4. The bad news is, this doesn’t work with Puppet agent 4.3.0.

Here’s how I found out that provider => init doesn’t work with Puppet agent 4.3.0:

# /opt/puppetlabs/puppet/bin/puppet resource service cups provider=init --debug
Debug: Runtime environment: puppet_version=4.3.0, ruby_version=2.1.7, run_mode=user, default_encoding=UTF-8
Debug: Evicting cache entry for environment 'production'
...
Debug: Puppet::Type::Service::ProviderInit: false value when expecting true
Error: /Service[cups]: Provider init is not functional on this host
Debug: Finishing transaction 40939920
Debug: Storing state
Debug: Stored state in 0.05 seconds
Debug: Puppet::Type::Service::ProviderInit: false value when expecting true
Error: Could not run: Provider init is not functional on this host

Fortunately, I found a common provider which works on both older and new version of Puppet agent. It’s called ‘debian’!

# /opt/puppetlabs/puppet/bin/puppet resource service cups provider=debian --debug
...
Debug: Executing: '/etc/init.d/cups status'
Debug: Finishing transaction 17147680
Debug: Storing state
Debug: Stored state in 0.05 seconds
Debug: Executing: '/etc/init.d/cups status'
service { 'cups':
  ensure => 'running',
}

Here’s a revised cups service manifest:

class cups::service {
  # [2015-12-16:SS] There has been an issue with upstart service
  #                 starting cups. Sometimes it reports cups is running,
  #                 when it in fact is not.
  #                 In Puppet, we can explicitly tells services to use
  #                 different startup script. In this case, 'init.d'.
  service { 'cups':
    ensure     => running,
    enable     => true,
    hasstatus  => true,
    hasrestart => true,
    provider   => debian,
  }

}

Well, that’s it. If you have any comments or suggestions, do let me know.

Disable puppet-lint class inheriting from params class

I normally use Vim to edit Puppet modules and manifests codes. However, recently I just found out about Atom editor, and quite like it. Out of the box, Atom supports many programming languages by providing syntax highlight. There are hundreds of Packages available which provide more functionalities such as make Atom to behave like a Vi editor, support Puppet codes, etc.

As mentioned earlier, Atom provides syntax highlighting for various programming languages; well except Puppet. So we need to manually install a package that fills that requirement: language-puppet. I also suggest that you additionally install linter and linter-puppet packages.

By default puppet-lint (provided by linter-puppet package), will complain about the use of class inheriting because this feature is not compatible with Puppet 2.6.2 and earlier. However, in my case I’m using Puppet 3.8.1, and I really like class inheriting.

To disable that puppet-lint warning, you can just pass this option to the puppet-lint command line:

--no-class_inherits_from_params_class-check

Well, I’m not using puppet-lint directly, and puppet-lint is called by the Atom editor. Here’s how it can be disabled:

Edit ~/.atom/config.cson:

  "linter-puppet-lint":
    puppetLintArguments: "--no-autoloader_layout-check --no-class_inherits_from_params_class-check"
    puppetLintExecutablePath: "/usr/bin/puppet-lint"

Ref: http://puppet-lint.com/checks/class_inherits_from_params_class/

របៀបលុបណូត​ចេញ​ពី Puppet master

នៅ​កន្លែង​ធ្វើ​ការ​ ខ្ញុំ​ប្រើ​ Puppet ដើម្បី​គ្រប់​គ្រង​ ម៉ាស៊ីន​ជា​ច្រើន។ ជួនកាល​ខ្ញុំ​ត្រូ​វការលុប​ម៉ាស៊ីន​ចេញ ពី Puppet master ដោយ​សារ​ម៉ាស៊ីន​នោះ​លែង​ត្រូវ​ការហើយ។ នេះ​ជា​ជំហាន​ដែល​ត្រូវ​ធ្វើ៖

១. លុបមាស៊ីន​ ឬ ណូត​នោះ​ពី puppet dashboard
២. លុប certificate របស់ម៉ាស៊ីននោះ​ចេញពី puppet master ដោយវាយ ខំម៉ាន

# puppet cert clean node123.example.com

៣. ដោយសារ​ខ្ញុំ​ប្រើ PuppetDB ខ្ញុំ​ត្រូវ​លុប​ ណូត​នោះ ចេញ​ពី PuppetDB ដែរ

# puppet node deactivate node123.example.com

ខ្ញុំ​សរសេរ អត្ថបទ​នេះ ដើម្បី​រំលឹក​ខ្លួន​ឯង​ពេល​ខ្ញុំ​ត្រូវ​ការ​ធ្វើ​វា​ម្តង​ទៀត។ បើ​សិន​ជា​លោកអ្នក​ មាន​សំណូម​ពរ​ ឬ​សំណួរ​ទាក់ទង​នឹង​ ការប្រើប្រាស់ Puppet អាចសួរ​ខ្ញុំ​បាន៕

Puppet agent hangs after loading facts

At my current work place, we use Puppet to manage desktop machines. Recently, I need to prepare an new image based on Ubuntu 14.04 LTS.

To prepare an image, in brief, first thing is to install a fresh Ubuntu on a machine, compile and package lots of applications, and make change to configuration such as look and feels, etc. As mentioned earlier, I use Puppet to manage machine, so I store all customed configuration on the Puppet server.

While working on this image, I came across a problem with Puppet agent, the software running on each desktop. The issue is that it took very long to execute the code. Here is an example:

matht336:~# puppet agent -vt -l console
Info: Loading facts in /var/lib/puppet/lib/facter/sku.rb
Info: Loading facts in /var/lib/puppet/lib/facter/users.rb
Info: Loading facts in /var/lib/puppet/lib/facter/zid.rb
Info: Loading facts in /var/lib/puppet/lib/facter/videocard.rb
Info: Caching catalog for matht336
Info: Applying configuration version '1398116230'
Notice: Finished catalog run in 1.09 seconds

According to the above output, it only took 1.09 seconds to run the catalog. Well the truth is it took way longer than that, approximately about 20 minutes. I re-ran puppet agent with -d flag for debug, but none of the output information gave me any clues of what could possibily go wrong.

One clue that I’ve been missing and should have noticed much earlier is that puppet agent got stuck after loading the facts.

So I modified /etc/puppet/puppet.conf on the node (not the server), to not receiving custom facts from the server:

[main]
...
pluginsync=false

Then, I removed the custom facts one by one from /var/lib/puppet/facter and re-run puppet agent on the node. Suprisingly, zid.rb was the source of the puppet agent slowness. zid.rb is a fact written by my colleague to keep track of who last logged in to a machine.

I’ve been wondering why noone on the Internet has the same issue as me. At first I thought maybe Ubuntu 14.04 is too and not many people have been running puppet on it. Now I know, it the problem is unique to my environment.

So if you experience the same problem as me, I would suggest that the first thing to check out is openning up thos custom facts and see what they’re doing.

How to fix PuppetDB SSL error

If you don’t know what PuppetDB is, this post probably will not be any useful to you. But if you wanna find out or learn about it, here is the link: http://docs.puppetlabs.com/puppetdb/latest/index.html.

For PuppetDB to work, we need to generate SSL certificates from a Puppet master’s SSL certificate. In my case, I need to re-configure this PuppetDB to a new Puppet mater which sits on the same server. Here’s how it was done:

1) Stop PuppetDB

# service puppetdb stop

2) Generate SSL certificates:

# puppetdb ssl-setup

3) Restart PuppetDB service:

# service puppetdb restart

As mentioned, the PuppetDB and Puppet master happen to be on the same server. If your Puppet master is on a different server, you’ll need to manually copy the SSL certificate of that mater to generate ones for PuppetDB. You can check this well-documented how to at: Manually Generating and Preparing Certificates.

Fix puppet-dashboard pending tasks

ផាប់ភិត​ដាសបត ឈប់​ដើរ​ ហើយ​ pending tasks មាន​ប្រ​ហែល​ជាង ២០០០។

# ps -ef | grep job
www-data  4799     1  0 14:07 ?        00:00:00 delayed_job.0_monitor
www-data  4804     1  0 14:07 ?        00:00:00 delayed_job.1_monitor
root      8300 31461  0 15:22 pts/0    00:00:00 grep job

មិន​ដឹង​ថា​ជាមាន​វិធី​ល្អ​ សំរាប់​ជួសជុល​បញ្ហា​នេះទេ ប៉ុន្តែខ្ញុំ​អាច ប្រើវិធីដូចនេះ

# cd /usr/share/puppet-dashboard/
# rake RAILS_ENV=production jobs:work

អោយវារត់​ យ៉ាងយូរ​ បន្ទាប់​មក puppet-dashboard-worker ក៏ចាប់ផ្តើម​ដើរវិញ

# ps -ef | grep job
www-data  4799     1  0 14:07 ?        00:00:00 delayed_job.0_monitor
www-data  4804     1  0 14:07 ?        00:00:00 delayed_job.1_monitor
www-data  5089     1 24 14:15 ?        00:20:43 dashboard/delayed_job.0
www-data  5094     1 24 14:15 ?        00:20:45 dashboard/delayed_job.1
root      8570 31461  0 15:41 pts/0    00:00:00 grep job