<!DOCTYPE html><html><head><meta charset="utf-8"><title>Slurm_Blog_Post.md</title><style></style></head><body>
<p>Whereas workstations can be quite powerful, there are a few scenarios demanding the need for separate machines linked together to enable batch-like processing through either MPI or simple scripting,</p>
<pre><code>1. A simple need for horsepower.
2. Resilient compute resources (via a large lack of human interaction).
3. Heavy webapp hosting.
4. Dev-ops.
</code></pre><p>Whether one decides to build a cluster by buying cloud compute time or putting something together themselves, nevertheless, a batch job scheduler and shared filesystem must be put in place for anything to work at all!</p>
<p>In the post, we show how to set up a common opensource job scheduler, SLURM, in addition to a shared filesystem via Gluster. SLURM supports GPUs and we have found through practice that whereas not the fastest filesystem, Gluster can take a lickin&#39; and keep tickin&#39; through some wholly unknown means.</p>
<h2 id="initial-cluster-installation">Initial Cluster Installation</h2>
<p>Our instructions assume that you have already installed Debian on a set of machines attached via IP. Moreover, we assume that each machine is named <code>c000</code>, <code>c001</code>, etc. and may be pinged as such, <code>ping c000</code>, through whatever means you see fit. We also assume each machine possesses a single <strong>CUDA</strong> GPU.</p>
<p>Two machines in the cluster must be treated specially as these will be Gluster servers. Install a small root partition (say, 30G) and leave the rest open with an empty directory at <code>/data/gv0/brick1</code>. A secondary disk may be preferred to mount at that point and a preferably equal amount of space should be available on each of the two machines.</p>
<h2 id="gluster">Gluster</h2>
<p>Practical work across a cluster requires a <em>stable</em> shared filesystem. Whereas NFS and AFS are much faster than Gluster, an overloaded NFS or AFS server can screem to a halt, hanging clients and doing even worse if they are rebooted! For safety, we tend to favor Gluster as in our practice no matter how we have abused it, it has never lead to cluster-wide failure.</p>
<p>In this post, we give quick instructions on how to set up a Gluster filesystem shared via two servers - - you can expand on the instructions to do more, but two servers seem fairly resilient and responsive.</p>
<p>Each machine will need gluster, current time keeping, and a place to mount the gluster filesystem as native and also NFS. Whereas the former leads to faster filesystem performance, NFS allows the use of cachefiles.d - - a boon for read-heavy jobs.</p>
<pre><code># gluster build req
apt-get install -y libfuse-dev flex bison libssl-dev xfsprogs build-essential

# NTP
apt-get install -y ntp
cat &gt;&gt; /etc/ntp.conf &lt;&lt;EOF
server 0.US.pool.ntp.org
server 1.US.pool.ntp.org
EOF

# binaries
wget http://download.gluster.org/pub/gluster/glusterfs/3.6/3.6.2/glusterfs-3.6.2.tar.gz -Oglusterfs.tar.gz
tar xzfvp glusterfs.tar.gz
(
  cd glusterfs-3.6.2
  ./configure
  make install
)

# mount points for the filesystem
mkdir -p /mnt/gfs /mnt/nfs
</code></pre><p>Assuming you named your file servers <code>c000</code> and <code>c001</code>, set up the gluster daemon to start on boot,</p>
<pre><code># bring up gluster server on boot
update-rc.d glusterd defaults
echo /usr/local/lib &gt;&gt; /etc/ld.so.conf
ldconfig
service glusterd start
</code></pre><p>Get each server talking to each other,</p>
<pre><code>gluster peer probe c001        # &lt;- on c000
gluster peer probe c000        # &lt;- on c001
</code></pre><p>On <code>c000</code>, create a volume mirrored across both servers. Note the reference to <code>/data/gv0/brick1</code> as explained in the previous section - - fileserver data is stored here.</p>
<pre><code>gluster volume create gv0 replica 2 c000:/data/gv0/brick1 c001:/data/gv0/brick1 force
gluster volume start gv0
gluster volume info
</code></pre><p>On the remaining workers in your cluster you&#39;ll need a <a href="http://blog.gluster.org/tag/debian-2/">hack</a> to make sure that they all mount native Gluster on boot,</p>
<pre><code># mount gluster on boot
cat &gt;&gt; /etc/fstab &lt;&lt;EOF
c000:/gv0 /mnt/gfs glusterfs defaults,_netdev 0 0
EOF

# make sure that fuse is running before local fs mount
sed -i &#39;s/# Required-Start:[ \t]+$remote_fs/# Required-Start:    $local_fs/&#39;
update-rc.d fuse start 34 S . stop 41 0 6 .
</code></pre><p>You&#39;ll also want to enable <code>cachefilesd</code> via an NFS mount to cache files locally as they are read across the network. An arbitrary amount of data is then cached clientside making I/O reads much faster and keeping load off of the fileservers,</p>
<p>  sudo apt-get install cachefilesd
  sed -i &#39;/RUN/ d&#39; /etc/default/cachefilesd
  echo &#39;RUN=yes&#39; &gt;&gt; /etc/default/cachefilesd
  service cachefilesd start   # this will fail, but it is working if <code>ps ax | grep cachefiles</code> says its running</p>

</body></html>