Category Archives: server

rsyncd won’t bind problem, determine what pid uses a port

We had a problem with one of our server. It’s rsyncd is not responding anymore. It’s listening to the port but it’s not accepting requests.

Here’s what the log says:

[root@SERVER ~]# tail /var/log/messages
Jun 21 13:19:46 SERVER xinetd[28270]: Swapping defaults
Jun 21 13:19:46 SERVER xinetd[28270]: readjusting service amanda
Jun 21 13:19:46 SERVER xinetd[28270]: bind failed (Address already in use (errno = 98)). service = rsync
Jun 21 13:19:46 SERVER xinetd[28270]: Service rsync failed to start and is deactivated.
Jun 21 13:19:46 SERVER xinetd[28270]: Reconfigured: new=0 old=1 dropped=0 (services)
Jun 21 13:21:34 SERVER xinetd[28270]: Exiting...
Jun 21 13:22:09 SERVER xinetd[32476]: bind failed (Address already in use (errno = 98)). service = rsync
Jun 21 13:22:09 SERVER xinetd[32476]: Service rsync failed to start and is deactivated.
Jun 21 13:22:09 SERVER xinetd[32476]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
Jun 21 13:22:09 SERVER xinetd[32476]: Started working: 1 available service

We tried stopping xinetd but there is still a process bound to the 873 port:

[root@SERVER ~]# service xinetd stop
Stopping xinetd: [ OK ]
[root@SERVER ~]# telnet localhost 873
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
^]
telnet> quit
Connection closed.

If only we could determine what process is still bound to the 873 port…

Well, there’s an app for that: lsof -i tcp:<port>

[root@SERVER ~]# lsof -i tcp:873
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
rpc.statd 1963 rpcuser 7u IPv4 4798 TCP *:rsync (LISTEN)
[root@SERVER ~]# kill 1963
[root@SERVER ~]# kill 1963
-bash: kill: (1963) - No such process
[root@SERVER ~]# telnet localhost 873
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
telnet: Unable to connect to remote host: Connection refused

Now that the process is dead, we restarted xinetd…

[root@SERVER ~]# service xinetd start
Starting xinetd: [ OK ]
[root@SERVER ~]# tail /var/log/messages
Jun 21 13:21:34 SERVER xinetd[28270]: Exiting...
Jun 21 13:22:09 SERVER xinetd[32476]: bind failed (Address already in use (errno = 98)). service = rsync
Jun 21 13:22:09 SERVER xinetd[32476]: Service rsync failed to start and is deactivated.
Jun 21 13:22:09 SERVER xinetd[32476]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
Jun 21 13:22:09 SERVER xinetd[32476]: Started working: 1 available service
Jun 21 13:23:06 SERVER xinetd[32476]: Exiting...
Jun 21 13:25:18 SERVER rpc.statd[1963]: Caught signal 15, un-registering and exiting.
Jun 21 13:25:18 SERVER portmap[3556]: connect from 127.0.0.1 to unset(status): request from unprivileged port
Jun 21 13:25:31 SERVER xinetd[3912]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
Jun 21 13:25:31 SERVER xinetd[3912]: Started working: 2 available services

… and that solves the problem. 🙂

References:

Advertisements

Install a MySQL NDB Cluster using CentOS 6.2 with 2 MGMs/MySQL servers and 2 NDB nodes

I wrote a post few weeks back that my MySQL NDB cluster was already running. This is a follow-up post on how I did it.

Before I dug in, I read some articles first on the best practices for MySQL Cluster installations. One of the sources that I’ve read is this quite helpful presentation.

The plan was to setup the cluster with 6 components:

  • 2 Management nodes
  • 2 MySQL nodes
  • 2 NDB nodes

Based on the best practices, I only need 4 servers to accomplish this setup. With these tips in mind, this is the plan that I came up with:

  • 2 VMs (2 CPUs, 4GB RAM, 20GB drives ) – will serve as MGM nodes and MySQL servers
  • 2 Supermicro 1Us (4-core, 8GB RAM, RAID 5 of 4 140GB 10k rpm SAS) – will serve as NDB nodes
  • all servers will be installed with a minimal installation of CentOS 6.2
The servers will use these IP configuration
  • mm0 – 192.168.1.162 (MGM + MySQL)
  • mm1 – 192.168.1.211 (MGM + MySQL)
  • lbindb1 – 192.168.1.164 (NDB node)
  • lbindb2 – 192.168.1.163 (NDB node)

That’s the plan, now to execute…

I downloaded the binary packages from this mirror. If you want a different mirror, you can choose from the main download page. I only need these two:

To install the packages, I ran these commands in the respective servers

    mm0> rpm -Uhv --force MySQL-Cluster-server-gpl-7.2.5-1.el6.x86_64.rpm
    mm0> mkdir /var/lib/mysql-cluster
    mm1> rpm -Uhv --force MySQL-Cluster-server-gpl-7.2.5-1.el6.x86_64.rpm
    mm1> mkdir /var/lib/mysql-cluster
    lbindb1> rpm -Uhv --force MySQL-Cluster-server-gpl-7.2.5-1.el6.x86_64.rpm
    lbindb1> mkdir -p /var/lib/mysql-cluster/data
    lbindb2> rpm -Uhv --force MySQL-Cluster-server-gpl-7.2.5-1.el6.x86_64.rpm
    lbindb2> mkdir -p /var/lib/mysql-cluster/data

The mkdir commands will make sense in a bit…

My cluster uses these two configuration files:

  • /etc/my.cnf  – used in the NDB nodes and MySQL servers (both mm[01] and lbindb[01])
  • /var/lib/mysql-cluster/config.ini – used in the MGM nodes only (mm[01])

Contents of /etc/my.cnf:

[mysqld]
# Options for mysqld process:
ndbcluster # run NDB storage engine
ndb-connectstring=192.168.1.162,192.168.1.211 # location of management server

[mysql_cluster]
# Options for ndbd process:
ndb-connectstring=192.168.1.162,192.168.1.211 # location of management server

Contents of /var/lib/mysql-cluster/config.ini:

[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2 # Setting this to 1 for now, 3 ndb nodes
DataMemory=1024M # How much memory to allocate for data storage
IndexMemory=512M
DiskPageBufferMemory=1048M
SharedGlobalMemory=384M
MaxNoOfExecutionThreads=4
RedoBuffer=32M
FragmentLogFileSize=256M
NoOfFragmentLogFiles=6

[ndb_mgmd]
# Management process options:
NodeId=1
HostName=192.168.1.162 # Hostname or IP address of MGM node
DataDir=/var/lib/mysql-cluster # Directory for MGM node log files

[ndb_mgmd]
# Management process options:
NodeId=2
HostName=192.168.1.211 # Hostname or IP address of MGM node
DataDir=/var/lib/mysql-cluster # Directory for MGM node log files

[ndbd]
# lbindb1
HostName=192.168.1.164 # Hostname or IP address
DataDir=/var/lib/mysql-cluster/data # Directory for this data node's data files

[ndbd]
# lbindb2
HostName=192.168.1.163 # Hostname or IP address
DataDir=/var/lib/mysql-cluster/data # Directory for this data node's data files

# SQL nodes
[mysqld]
HostName=192.168.1.162

[mysqld]
HostName=192.168.1.211

Once the configuration files are in place, I started the cluster with these commands (NOTE: Make sure that the firewall was properly configured first):

mm0> ndb_mgmd --ndb-nodeid=1 -f /var/lib/mysql-cluster/config.ini
mm0> service mysql start
mm1> ndb_mgmd --ndb-nodeid=2 -f /var/lib/mysql-cluster/config.ini
mm1> service mysql start
lbindb1> ndbmtd
lbindb2> ndbmtd

To verify if my cluster is really running, I logged-in into one of the MGM nodes and ran ndb_mgm like this:

I was able to set it this up a few weeks back. Unfortunately, I haven’t had the chance to really test it with our ETL scripts… I was occupied with other responsibilities…

Thinking about it now, I may have to scrap the whole cluster and install a MySQL with InnoDB + lots of RAM! hmmm… Maybe I’ll benchmark it first…

Oh well… 🙂

References:

Release the beast! Xeon X5650 + RAID 0 Intel 710 SSD (2)

Well… not really a beast in today’s standards… but it’s a beast if we’re talking about processor blades for a 10-blade Supermicro enclosure. Supermicro released its Sandy Bridge Xeons but those are available for a 14-blade enclosure for now.

I wrote a post before about our test bed machine to figure out if SSD is a good route for us. We concluded that it is so we opted to go with the enterprise offerings of Intel, the 710 series.

The Intel 710 SSD is really not inexpensive. I decided to go with RAID 0 of 2 Intel 710 200GB because of 2 reasons: (1) to increase space & (2) to distribute writes for (much) longer write endurance.

Technical specs:

  • Supermicro SBI-7126T-S6
  • 2 x Intel Xeon X5650
  • 2 x Intel 710 200G SSD (RAID0)
  • 8 x 8 GB DDR3

The X5650 is a 6-core CPU but because of Hyper-threading, it’s registering as 12 cores.

The new blade was deployed as a ganeti node in one of our production clusters. So far, the 2 Xeon X5650s are doing well even with 8 CPU-hungry virtual machines.

I configured the server with growth in mind, thus the 64G memory. We’re not using that much RAM right now but that will change quickly once the ganeti node will host DB servers.

I was planning to perform some benchmarks once we deployed it but I haven’t got the chance… Hopefully, I’ll have the time when the next one arrives. 🙂

Set-up a headless “file server”/”file burning station” with CentOS/RHEL 5 using VNC

I’m setting up a file server for my team that has to minimally comply with the following:

  • it has to cost NOTHING, parts have to be salvaged from old servers
  • it has to have a disk space of at least 1TB (must be RAID for good read/write performance)
  • it has to be headless (no monitor, keyboard and mouse), with GUI and accessible remotely
  • it has to have the ability to burn files using a USB DVD writer

setting up the disk space

I used (4) 500GB SATA drives and (2)  250 PATA drives (salvaged from our old servers) for this one. Deploying software RAID in CentOS is easy so I won’t elaborate on that.

Suffice to say my software RAID has this configuration:

going headless using VNC

I installed the GNOME desktop by default and accessed it remotely using VNC. Here’s a good guide [wiki.centos.org] on how to do it. Everything went well (at first…)

I hit a snag when I unplugged the monitor. After a reboot, the X server won’t load anymore. Last few lines of  /var/log/Xorg.0.log says “No screens found”. Apparently, installing the proprietary Nvidia driver has a side effect. The driver will try to “auto-detect” the monitor. But since no monitor is attached, the X server won’t load! (duh! so much for going headless…)

Reverting to the open-source driver fixed the problem. I opened /etc/X11/xorg.conf and changed “nvidia” to “nv“, then restarted X server.

burning files in CentOS?

Now, this one should be easy right. Well… apparently not… This got me baffled at first because I can’t access the DVD writer plugged in the system (got spoiled with Ubuntu). Eventually, I figured that it’s a file permission problem. The user must be added to the disk group. I added my username to the disk group by running this command as root:

$ usermod -aG disk pro

After a reboot I can now access the writer using k3b 🙂

Here’s a screenshot of the headless file server. I’m accessing it using Vinagre from my laptop running Ubuntu 10.04