As I’ve shared in my previous post, I was working on hardening AWS Linux 2014.09 using Puppet. I got that finished and I’ll probably improve the class in the future to make it easier to manage.
The first thing you’ll realize when you harden a system is this: it will break stuff… No, I’m not talking about the small stuffs, I’m talking about major applications that you rely on.
The first to break that I noticed was running a Puppet master with httpd + mod_passenger in AWS Linux 2014.09. It won’t work properly anymore… Ugh!
Since I was in hurry when I first discovered this, I just reverted back to the default Puppet master using Ruby webrick. Performance was really not an issue since this is only for our environment where we write/test our Puppet classes…
But when I decided to separate the CA server of our Puppet master for scalability — I can’t use the default Ruby webrick anymore. I have to allot the time and investigate if what’s the root cause of the problem
snapshot of the error
Clearly… it has something to do with permissions. The web server (httpd) cannot access the Unix socket created by mod_passenger.
Reviewing the CIS guidelines for AWS Linux 2014.09 pointed me to “3.1 Set Daemon umask” which states:
This is enforced by this Puppet class via this sysconfig configuration. CIS 3.1 guideline clearly says that “The daemon process can manually override these settings if these files need additional permission.” — which gave me an idea to override this in the httpd level.
So… override we go…
I opened /etc/init.d/httpd in vim and added a less strict umask: umask 0022
Restarted httpd and then… FIXED! 🙂
We have this big website that’s currently being overhauled (means: new architecture, new tech stack and totally new code from the ground up). The lead dev asked our team if we can redirect traffic to a static site in case the actual site is down.
[Update: Our site is now launched! It’s still in beta. Check it out here: https://new.smartnet.ph]
I only overheard this but I jumped in to help because I’ve been wanting to try this feature of Route 53 but didn’t have the chance to really implement it.
I figured that that there should be a lot of tutorials on how to do this already… so this should be a walk in the park.
A little help from Google lead me to a few sites. This one is a good tutorial if you only want to redirect to different IP (steps are listed and screenshots!).
I didn’t find a good tutorial as far as aliases are involved. And we’re stuck with this loading screen:
Not really a walk in the park…
With that good tutorial as reference, we (with help from John) decided to have a crack at this ourselves.
Note: This guide assumes that your domain is already hosted in Route 53, if not you must move it first.
This how we did it:
- create a static site hosted in S3 [how?] – skip Step 3
- create your route 53 health checks [how?] – replace Step 8 with the steps below
Create a secondary alias failover using AWS CLI:
- get the Hosted zone ID of your S3 endpoint [here] – In our case we’re using Singapore so hosted id is
- get the Hosted zone ID of your domain [how?] – in this guide, let’s assume that
mysite.ph has a zone id of
- create a json file like below:
serenity:~ deadlockprocess$ cat ~/tmp/mysite.ph.json
"Comment": "mysite.ph failover",
- add the failover alias as a new record set in Route 53 with this command:
serenity:~ deadlockprocess$ aws route53 change-resource-record-sets --hosted-zone-id ABCDE12345 --change-batch file:///Users/deadlockprocess/tmp/mysite.ph.json
- you can now go back to this guide and do Step 9 onwards
- also, allow the Route 53 Health Checkers’ IPs in your firewall/security group