Main

h1. Server Setup

In late 2012 I decided it was time to retire the old Dell desktop (the original phalanx) that had been limping along as my fileserver and build a proper fileserver.  The new [[wiki:/CustomComputers/2012/Phalanx|phalanx]] was born. Here are some of the important design/configuration decisions that I made:

* boot drive is a small SSD that is separate from the storage array which allows OS to be upgraded independently of file storage

* software RAID 5 (device mapper raid) using 6 X 3TBB HDDs (12 TB raw, ~11.1 TB useable)

Although phalanx's primary purpose is as an internal fileserver, it runs 24/7 and has plenty of spare cycles to also act as a webserver. Below are some of the things it is currently running.

h2. Update History

*7/8/2019*

   * Change Dynamic DNS provider to http://freedns.afraid.org. 

*1/9/2019*

  * Added 10G Ethernet interface.

*1/4/2019*

  * Replaced mobo/proc/RAM with parts salvaged from work. Proc i7-4790 (Haswell); RAM 32GB DDR3L.

*6/2/2018*

  * Upgraded boot drive to 500 GB SSD. Also installed Ubuntu 18.04 LTS.*

*1/16/2017*

  * Upgraded server added 2 x 3 TB additional HDDs (via PCIe SATA expansion card), new power supply, and new (slightly used) case.  After adding the new disks to the RAID and growing the file system, total usable storage on the array is now 11.18 TB.

   Parts List:

   ** 2 - Seagate 3TB BarraCuda SATA (ST3000DM008) 3.5-Inch Internal Hard Drive ($89.99 each)

   ** IO Crest SI-PEX40062 4 Port SATA III PCIe 2.0 X2 Controller Card ($37.99)

   ** SeaSonic G Series 550-Watt 80 PLUS GOLD Certified Modular Active PFC Power Supply SSR-550RM ($69.99)

   ** Corsair case (scavenged for free)

  * Instructions for adding disks, growing the array, and then growing the file system are below.

*11/18/2016*

  * Updated (wipe and reinstall) to Ubuntu Server 16.04.1 LTS

*9/2015*

   * I recently decided to pay for dynamic DNS service from "Dyn":http://dyn.com.  The cost was $30 (after 25% coupon) for a year w/ 30 hostnames. Server is now reachable at https://baranovich.homelinux.org:3268

*8/2014*  

 * Both of the my original 3TB HDDs started acting funny and died within a week of each other after about 1.8 years of continuous use.  Fortunately I reacted immediately and I was able to back up all of my data onto other disks. Although I regularly back up all of my critical data I decided that I really don't want to have to deal with losing any of my data.  I promptly purchased another HDD and a HW RAID card (an LSI MegaRAID SAS 9240-8i) and set out to build a RAID 5 array.  I quickly ran into problems with the RAID card.  I was only able to get the computer to boot with it installed once.  During that boot I upgraded the card's firmware with the hope that it would fix all my issues, but it did not and I was never able to fully boot with the card installed again.  After a little googling I discovered someone in my same position who claimed to have called LSI tech support and was told that LSI cards are "not compatible with newer motherboards".  The guy didn't elaborate as to what that meant but I assume it means LSI cards don't work in UEFI motherboards. I was disappointed but I started reading about software raid (device mapper raid) and decided that it should work perfectly for my little fileserver.  I also decided that btrfs is not really ready for prime time and I should use some other file system.

h2. OS Configuration

Current (November 2016) OS information:

| *Ubuntu Version:* | Ubuntu 18.04 Server LTS |

| *Kernel Version:* | 4.15.0-XX |

h2. DNS

Current hostname is: https://phalanx.homelinuxserver.org

To configure automatic DNS updates, make sure to install @ddclient@:

<pre>

$ sudo apt-get install ddclient

</pre>

Then follow instructions to log into account and select correct hostname.

h2. Network Interfaces

|*Interface*        | *MAC*           | *Reserved IP*        | *Notes*  |

|enp4s0             |d0:50:99:51:75:a4|192.168.8.6           |primary interface gigabit|

|enp5s0             |d0:50:99:51:75:a5|                      |secondary gigabit (unused)|

|enp1s0f0           |0c:c4:7a:1f:00:ae|192.168.250.3 (static)|10G fiber interface; direct to enthoo|

|enp1s0f1           |0c:c4:7a:1f:00:af|192.168.250.1 (static)|10G fiber interface; direct to brutus|

|ipmi               |bc:5f:f4:fe:6b:cb|192.168.208.1         |IPMI interface|

h2. Storage Configuration 

The software RAID5 array is block device @/dev/md0@ which is mounted at @/storage@; total usable storage is about -5.5TB- 11.1TB with the current drive configuration

The primary mechanism for accessing the storage on phalanx is via NFS v4. I have found that performance is excellent (much better than samba!) and NFS is supported fairly well on both Mac OS and Linux, which are the main client OSes I use.  The only annoyance I discovered is that uids on all systems MUST match.  This is no big deal on Linux, but I had to manually change my uid on Mac OS, which was not as easy as it should have been. 

h3. Storage setup

*NOTE:* After installing Ubuntu 16.04.1 I was pleasantly surprised to discover that none of the steps below were necessary.  The software RAID had already been found and assembled for me.  The stuff below is for reference.

* Install mdadm:

<pre>

$ sudo aptitude install mdadm xfsprogs

</pre>

* For each drive (sdb, sdc, sdd):

<pre>

# parted -a optimal /dev/sdb

GNU Parted 2.3

Using /dev/sdb

Welcome to GNU Parted! Type 'help' to view a list of commands.

(parted) mklabel gpt

Warning: The existing disk label on /dev/sdd will be destroyed and all data on

this disk will be lost. Do you want to continue?

Yes/No? y

(parted) mkpart primary 0% 100%

(parted) align-check opt 1

1 aligned

(parted) quit

Information: You may need to update /etc/fstab.

</pre>

* Create the array using mdadm:

<pre>

# mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sd[b-d]1

</pre>

* The above step takes quiet a while, about 6 hours in my case. To watch progress:

<pre>

$ watch cat /proc/mdstat

</pre>

* Once this is done you can create the XFS file system on the new @/dev/md0@ block device:

<pre>

$ sudo mkfs -t xfs -d su=512k -d sw=2 /dev/md0

</pre>

* To make sure the array reappears after reboot:

<pre>

# mdadm --examine --scan >> /etc/mdadm/mdadm.conf  # NOTE: this syntax is deprecated and no longer works!

# mdadm --detail --scan >> /etc/mdadm/mdadm.conf

# update-initramfs -u

</pre>

* If any issues arise you can stop a software RAID and reassemble it using:

<pre>

$ sudo mdadm --stop /dev/md127

$ mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1

</pre>

* To view array status:

<pre>

dsorber@phalanx:~$ sudo mdadm --detail /dev/md0

/dev/md0:

        Version : 1.2

  Creation Time : Fri Aug 22 17:05:41 2014

     Raid Level : raid5

     Array Size : 11720534016 (11177.57 GiB 12001.83 GB)

  Used Dev Size : 2930133504 (2794.39 GiB 3000.46 GB)

   Raid Devices : 5

  Total Devices : 5

    Persistence : Superblock is persistent

    Update Time : Fri Jan 20 16:45:45 2017

          State : clean 

 Active Devices : 5

Working Devices : 5

 Failed Devices : 0

  Spare Devices : 0

         Layout : left-symmetric

     Chunk Size : 512K

           Name : phalanx:0  (local to host phalanx)

           UUID : 54bfb945:47ec32da:d2ccc4db:9d3615fa

         Events : 4001

    Number   Major   Minor   RaidDevice State

       0       8       17        0      active sync   /dev/sdb1

       1       8       33        1      active sync   /dev/sdc1

       3       8       49        2      active sync   /dev/sdd1

       5       8       81        3      active sync   /dev/sdf1

       4       8       65        4      active sync   /dev/sde1

</pre>

* @/etc/fstab@

<pre>

# /etc/fstab: static file system information.

#

# Use 'blkid' to print the universally unique identifier for a

# device; this may be used with UUID= as a more robust way to name devices

# that works even if disks are added and removed. See fstab(5).

#

# <file system> <mount point>   <type>  <options>       <dump>  <pass>

# / was on /dev/sda1 during installation

UUID=66e8057c-1b75-40ba-8ceb-480b6e4405ab /               ext4    errors=remount-ro 0       1

/dev/md0        /storage        xfs     rw,nobarrier,noatime,nodiratime 0       0

/storage        /export/storage    none    bind    0    0

</pre>

h3. NFS setup:

* Install NFS (note additional settings below under firewall configuration):

<pre>

$ sudo apt install nfs-kernel-server

</pre>

* Edit @/etc/fstab@ and add to mount @/storage@ on @/exports/storage@:

<pre>

/storage	/export/storage    none    bind    0    0

</pre>

* Edit @/etc/exports@:

<pre>

/export            10.0.0.0/24(rw,fsid=0,insecure,no_subtree_check,async)

/export/storage    10.0.0.0/24(rw,nohide,insecure,no_subtree_check,async)

</pre>

* Restart NFS:

<pre>

$ sudo systemctl restart nfs-kernel-server

</pre>

h2. Firewall Configuration

|= *Port* =|= *Description*=|

|22|ssh|

|80|http|

|111|SUN rpc|

|443|https|

|2049|NFS|

|3142|apt-cacher-ng|

|4045|lockd|

|17641|NFS connect|

|61993|bt|

It was a bit annoying to get NFS working when the firewall was enabled but I found some good instructions and got them to work.  The issue was that some of the important rpc/NFS daemons use random ports when they boot up.

* Edit @/etc/default/nfs-kernel-server@ comment out this line:

<pre>

RPCMOUNTDOPTS=--manage-gids

</pre>

 add this line

<pre>

RPCMOUNTDOPTS="--port 17641"

</pre>

* Edit/create @/etc/modprobe.d/options.conf@:

<pre>

options lockd nlm_udpport=4045 nlm_tcpport=4045

</pre>

* Edit @/etc/modules@ and add @lockd@ on its own line:

<pre>

...

lockd

</pre>

* Now create the firewall rules (only allow incoming access for the local network), for each port in the table above:

<pre>

$ sudo ufw allow from 10.0.0.0/24 to any port <PORT>

</pre>

* Add external allow rules (i.e. up these ports to the outside world) for ssh, https, and bt :

<pre>

$ sudo ufw allow <PORT>

</pre>

* Start ufw:

<pre>

$ sudo ufw enable

</pre>

Overview of added rules:

<pre>

sudo ufw allow proto tcp to 0.0.0.0/0 port 22

sudo ufw allow proto tcp to 0.0.0.0/0 port 443

sudo ufw allow proto tcp to 0.0.0.0/0 port 61993

sudo ufw allow from 192.168.4.0/24 to any port 111

sudo ufw allow from 192.168.4.0/24 to any port 2049

sudo ufw allow from 192.168.4.0/24 to any port 3142

sudo ufw allow from 192.168.4.0/24 to any port 4045

sudo ufw allow from 192.168.4.0/24 to any port 17641

sudo ufw allow from 192.168.8.0/24 to any port 111

sudo ufw allow from 192.168.8.0/24 to any port 2049

sudo ufw allow from 192.168.8.0/24 to any port 3142

sudo ufw allow from 192.168.8.0/24 to any port 4045

sudo ufw allow from 192.168.8.0/24 to any port 17641

sudo ufw allow from 192.168.20.0/24 to any port 3142

sudo ufw allow from 192.168.250.0/24 to any port 111

sudo ufw allow from 192.168.250.0/24 to any port 2049

sudo ufw allow from 192.168.250.0/24 to any port 4045

sudo ufw allow from 192.168.250.0/24 to any port 17641

</pre>

h2. Apache

I didn't originally document my Apache setup, but when wiping to install 14.04 I remedied this. The [[source:Default Repo/configs/apache|Apache configs]], are the really important part here. Below is a section on each application that's running on/from Apache.  I am using https with a permanent redirect from port 80 along with a self-signed certificate.  Most of the sections below are password protected using mod_auth_digest.  The appropriate digest file is created and managed with the "htdigest":http://httpd.apache.org/docs/2.2/programs/htdigest.html tool. *NOTE:* make sure the password digest file (e.g. @/home/dsorber/web_users@) is owned by @www-data:www-data@ and permissions are set to @600@.

* Install apache2 itself:

<pre>

$ sudo aptitude install apache2

</pre>

* Enable apache modules:

<pre>

$ sudo a2enmod ssl proxy proxy_http auth_digest

</pre>

* Copy backed up apache configs to their location:

<pre>

$ sudo cp apache_configs/* /etc/apache2/sites-available/

</pre>

* Setup SSL cert:

<pre>

cd /etc/apache2

sudo mkdir ssl

sudo cp baranovich.{crt,key} /etc/apache2/ssl

</pre>

* Turn on main site (also disable default apache2 site):

<pre>

$ sudo a2ensite main

</pre>

Overview of rules to add:

<pre>

sudo ufw allow proto tcp to 0.0.0.0/0 port 22

sudo ufw allow proto tcp to 0.0.0.0/0 port 443

sudo ufw allow proto tcp to 0.0.0.0/0 port 61993

sudo ufw allow from 192.168.4.0/24 to any port 111

sudo ufw allow from 192.168.4.0/24 to any port 2049

sudo ufw allow from 192.168.4.0/24 to any port 4045

sudo ufw allow from 192.168.4.0/24 to any port 17641

sudo ufw allow from 192.168.8.0/24 to any port 111

sudo ufw allow from 192.168.8.0/24 to any port 2049

sudo ufw allow from 192.168.8.0/24 to any port 4045

sudo ufw allow from 192.168.8.0/24 to any port 17641

sudo ufw allow from 192.168.250.0/24 to any port 111

sudo ufw allow from 192.168.250.0/24 to any port 2049

sudo ufw allow from 192.168.250.0/24 to any port 4045

sudo ufw allow from 192.168.250.0/24 to any port 17641

</pre>

h3. Transmission

I use "Transmission":http://www.transmissionbt.com/|Transmission]] to download Linux .isos and give back to the community by sharing them using my own bandwidth. Transmission with its webui runs as a separate daemon so apache is configured to act as reverse proxy.

Transmission has its own [[source:Default Repo/configs/settings.json|configuration file]].

* Install transmission daemon:

<pre>

$ sudo aptitude install transmission-daemon

</pre>

* Copy over configuration:

<pre>

$ sudo systemctl stop transmission-daemon.service

$ sudo cp settings.json /etc/transmission-daemon

$ sudo systemctl start transmission-daemon.service

</pre>

* Make sure the user digest file is in place:

<pre>

$ cd /home/dsorber

$ cp web_users .

</pre>

* Enable site:

<pre>

$ sudo a2ensite transmission

</pre>

h3. Trac + Git

Make sure to backup the @trac@ folder and then install is a breeze. The trac projects live at @/home/dsorber/trac/projects@, there is currently only one project.

* Install git, mod_wsgi, and setuptools:

<pre>

$ sudo apt install git libapache2-mod-wsgi python-pip

</pre>

* Install trac:

<pre>

$ sudo pip install trac

</pre>

* Fix/adjust permissions:

<pre>

cd ~/trac/projects/headquarters

sudo chown -R www-data:www-data db/

sudo chown -R www-data:www-data log/

</pre>

* Enable site:

<pre>

$ sudo a2ensite trac

</pre>

* Also install Pygments for syntax highlighting:

<pre>

$ sudo pip install pygments

</pre>

After using Git a bit at work I decided that I'd give it a try for my personal use.  I'm still getting used to a few things but overall I like it.  Apparently I have configured git over ssh. My git repository clone path is:

<pre>

git clone /home/dsorber/trac/projects/headquarters/repo.git

</pre>

h3. Nextcloud

"Nextcloud":https://nextcloud.com/|Nextcloud is a self-hosted "cloud". It allows easy file viewing and sharing as well as !CardDav and !CalDav servers:

* Install necessary packages including MariaDB (MySQL):

<pre>

sudo apt install apache2 mariadb-server libapache2-mod-php7.2 php7.2-gd php7.2-json php7.2-mysql php7.2-curl php7.2-mbstring php7.2-intl php-imagick php7.2-xml php7.2-zip unzip

</pre>

* Download latest Nextcloud release (example) and unzip it:

<pre>

$ wget https://download.nextcloud.com/server/releases/nextcloud-13.0.2.zip

$ unzip nextcloud-13.0.2.zip

</pre>

* Move Nextcloud to webserver directory and change permissions:

<pre>

$ sudo mv nextcloud /var/www

$ sudo chown -R www-data:www-data /var/www/nextcloud/

</pre>

* Copy in apache config and enable necessary modules:

<pre>

$ cp nextcloud.conf /etc/apache2/sites-available/

$ sudo a2ensite nextcloud.conf 

$ sudo a2enmod rewrite headers env dir mime

$ sudo systemctl restart apache2

</pre>

* For some reason MariaDB does not setup an admin account when it is installed so you must do so manually. Keep this password in a safe place.

<pre>

$ sudo mysql --user=root mysql

% CREATE USER 'dbadmin'@'localhost' IDENTIFIED BY 'Apassword';

% GRANT ALL PRIVILEGES ON *.* TO 'dbadmin'@'localhost' WITH GRANT OPTION;

% FLUSH PRIVILEGES;

% exit

</pre>

* Now goto the web interface and enter the necessary information.

h4. Other configuration

* To setup local storage: go to Apps > Disabled Apps > External Storage (click Enable)

* To setup pretty URLS:

 * https://docs.nextcloud.com/server/13/admin_manual/installation/source_installation.html#pretty-urls

 * <pre> $ sudo vim /var/www/nextcloud/config/config.php </pre>

 * <pre> $ sudo systemctl restart apache2 </pre>

h2. Misc

* Setup the display script:

 * Edit @~/.profile@ and add @./repo/software/misc/display.py@

* Setup checker script:

 * 

<pre>

$ sudo su

# crontab -e

</pre>

 add this line:

<pre>

0 * * * *       /home/dsorber/repo/software/misc/checker.py

</pre>

* Install @apt-cacher-ng@:

<pre>

$ sudo apt install apt-cacher-ng

</pre>

 * Edit @/etc/apt/apt.conf.d/02proxy@ add: @Acquire::http::Proxy "http://192.168.8.6:3142";@

----

h3. Sources

* "Adding an extra disk to an mdadm array":http://zackreed.me/adding-an-extra-disk-to-an-mdadm-array/

* https://raid.wiki.kernel.org/index.php/Growing

* "Ubuntu Setting Up NFS HowTo":https://help.ubuntu.com/community/SettingUpNFSHowTo

* http://ubuntuforums.org/showthread.php?t=2228292

* http://www.mysolutions.it/tutorial-mdadm-software-raid-ubuntu-debian-systems/

* http://ubuntuforums.org/showpost.php?p=7959294&postcount=17

* http://trac.edgewall.org/wiki/TracModWSGI

* "Pro Git":http://git-scm.com/book

* http://baikal-server.com/

* https://github.com/jeromeschneider/Baikal

* https://easyengine.io/tutorials/linux/ubuntu-postfix-gmail-smtp/

* https://wiki.archlinux.org/index.php/Btrfs

* https://blog.packagecloud.io/eng/2015/05/05/using-apt-cacher-ng-with-ssl-tls/