Category Archives: puppet

[Puppet] fixing mod_passenger: Cannot connect to Unix socket – Permission denied in a hardened AWS Linux 2014.09

As I’ve shared in my previous post, I was working on hardening AWS Linux 2014.09 using Puppet. I got that finished and I’ll probably improve the class in the future to make it easier to manage.

The first thing you’ll realize when you harden a system is this: it will break stuff… No, I’m not talking about the small stuffs, I’m talking about major applications that you rely on.

The first to break that I noticed was running a Puppet master with httpd + mod_passenger in AWS Linux 2014.09. It won’t work properly anymore… Ugh!

Since I was in hurry when I first discovered this, I just reverted back to the default Puppet master using Ruby webrick. Performance was really not an issue since this is only for our environment where we write/test our Puppet classes…

But when I decided to separate the CA server of our Puppet master for scalability — I can’t use the default Ruby webrick anymore. I have to allot the time and investigate if what’s the root cause of the problem

passenger error

snapshot of the error

Clearly… it has something to do with permissions. The web server (httpd) cannot access the Unix socket created by mod_passenger.

versions

version information

Reviewing the CIS guidelines for AWS Linux 2014.09 pointed me to “3.1 Set Daemon umask” which states:

CIS 3.1

This is enforced by this Puppet class via this sysconfig configuration. CIS 3.1 guideline clearly says that “The daemon process can manually override these settings if these files need additional permission.”  — which gave me an idea to override this in the httpd level.

So… override we go…

I opened /etc/init.d/httpd in vim and added a less strict umask: umask 0022

httpd

Restarted httpd and then… FIXED! 🙂

[Puppet] The pesky error “`require’: no such file to load — json (LoadError)” when running puppet in AWS Linux

I encountered this error again when running puppet version 3.8.2 … I know I encountered this error before but I it took me a few minutes to remember how I solved it… so that I won’t forget in the future, I’m posting this here.

The error looks like this:

Screen Shot 2015-09-02 at 2.44.24 PM

To solve it, just run: yum install rubygem18-json.x86_64

Somebody forgot to include json gem as a dependency… 🙂

Link

[puppet forge] proletaryo-supervisor v0.4.0 now supports Ubuntu

Necessity is the great motivator.

I wrote this puppet module almost a year ago. The first version up to the last one only supports RedHat-based distros. Amazon Web Services is the primary platform that I use so the module is heavily tested and used in AWS Linux environments.

I was planning to support Ubuntu since day one but I managed to procrastinate because it’s not really needed in our deployments. That changed today though because we’re rolling out a few Ubuntu instances in AWS 🙂

I hope some people will find this module useful. It’s always open for contributions. Just fork it in GitHub:

https://github.com/proletaryo/puppet-supervisor

If you have an existing Puppet installation, just install it in your Puppet Server:

puppet module install proletaryo-supervisor –version 0.4.0

QuickFix: Cleaning up Puppet master’s report logs

OK… Our puppet master’s disk is slowly filling up with logs… We have around 70+ nodes already so no wonder it’s filling up at a high rate:

marionette-disk

I’ve known about the problem since 2 weeks ago but I always had an excuse to put it in my “next” to-do pile….

Well, today I can’t ignore it anymore since it’s almost full 🙂

So off to Google and do a little searching… And this is the best answer that I got:

tidy { "/var/lib/puppet/reports":
  age => "1w",
  recurse => true,
}

Duh! Use puppet to clean-up Puppet logs! *facepalm* (I felt stupid when I realized this…)

After knowing a new puppet type, I was excited to do a test run: puppet agent -t

Well… The ruby process ran for a few a minutes but it did nothing! Hmmm… I figured maybe because the number of files is staggering that’s why the puppet run can’t handle it. After all, I have >4GB worth of log files in /var/lib/puppet/reports/.

So a little help is needed then. I ran these commands to give puppet a hand:
find /var/lib/puppet/reports/ -type f -ctime +7 | xargs -P 4 -n 20 rm -f

And the next succeeding puppet runs worked like a charm 🙂

marionette-disk1

QuickFix: Can’t install puppetdb via puppet modules in AWS Linux

I was trying to install PuppetDB for my Puppet deployment using this guide.

And then when I ran puppet agent -t, my terminal was full of red error messages:

Notice: /Stage[main]/Postgresql::Initdb/Exec[postgresql_initdb]/returns: creating directory /var/lib/pgsql/data ... initdb: could not create directory "/var/lib/pgsql": Permission denied
Error: /usr/bin/initdb --encoding 'UTF8' --pgdata '/var/lib/pgsql/data' returned 1 instead of one of [0]
Error: /Stage[main]/Postgresql::Initdb/Exec[postgresql_initdb]/returns: change from notrun to 0 failed: /usr/bin/initdb --encoding 'UTF8' --pgdata '/var/lib/pgsql/data' returned 1 instead of one of [0]

My quick fix:
[root@-----]$ cd /var/lib/ && ln -s pgsql9/ pgsql