brutus Server Setup¶

History¶

Who needs "the cloud", I've got gigabit fiber running directly to my house!
-- Me

I've been running a local file server for 16ish years.

Date	Note
9/11/2024	Swtiched to Dynu for dynamic DNS and used Let's Encrypt for a proper SSL/TLS cert.
9/8/2024	Retired phalanx and replaced it with brutus (hardware details in the section below).
5/27/2022	Wipe and install Ubuntu 22.04
7/8/2019	Change Dynamic DNS provider to http://freedns.afraid.org.
1/9/2019	Added 10G Ethernet interface.
1/4/2019	Replaced mobo/proc/RAM with parts salvaged from work. Proc i7-4790 (Haswell); RAM 32GB DDR3L.
6/2/2018	Upgraded boot drive to 500 GB SSD. Also installed Ubuntu 18.04 LTS.
1/16/2017	Upgraded server added 2 x 3 TB additional HDDs (via PCIe SATA expansion card), new power supply, and new (slightly used) case. After adding the new disks to the RAID and growing the file system, total usable storage on the array is now 11.18 TB.
11/18/2016	Updated (wipe and reinstall) to Ubuntu Server 16.04.1 LTS
9/2015	I recently decided to pay for dynamic DNS service from Dyn. The cost was $30 (after 25% coupon) for a year w/ 30 hostnames. Server is now reachable at https://baranovich.homelinux.org:3268
8/2014	Both of the my original 3TB HDDs started acting funny and died within a week of each other after about 1.8 years of continuous use. Fortunately I reacted immediately and I was able to back up all of my data onto other disks. Although I regularly back up all of my critical data I decided that I really don't want to have to deal with losing any of my data. I promptly purchased another HDD and a HW RAID card (an LSI MegaRAID SAS 9240-8i) and set out to build a RAID 5 array. I quickly ran into problems with the RAID card. I was only able to get the computer to boot with it installed once. During that boot I upgraded the card's firmware with the hope that it would fix all my issues, but it did not and I was never able to fully boot with the card installed again. After a little googling I discovered someone in my same position who claimed to have called LSI tech support and was told that LSI cards are "not compatible with newer motherboards". The guy didn't elaborate as to what that meant but I assume it means LSI cards don't work in UEFI motherboards. I was disappointed but I started reading about software raid (device mapper raid) and decided that it should work perfectly for my little fileserver. I also decided that btrfs is not really ready for prime time and I should use some other file system.
??/2008	Created first version of phalanx with Dell desktop that was Kab's college computer.

Hardware Details¶

• Dell r730xd server 2U
  • 2x Intel E5-2680v4 CPUs
  • 128 GB DDR4 2400
  • 24x 1 TB mSATA PM-851
  • Dell HBA H330
  • LSI SAS 9300-8E 8-Port 12Gb/s 12Gbps External HBA Controller Bus Card
• Netapp DS2246 Storage Expansion Array 24 Bay 2.5" SAS Trays 2x IOM6 Controllers
  • 24x 1 TB mSATA PM-851

Total raw storage capacity: 44 TB
Total usable storage capacity (RAID-6): 42.84 TB

Price Breakdown¶

Components purchased from e-Bay.

Component	Description	Price
Dell r730xd server	2x Intel E5-2680v4 CPUs; 128 GB DDR4	$413.39
24x 2.5" 8FKXC/08FKXC	SAS SATA Drive Caddy For Dell PowerEdge	$88.79
Dell TRJ5G	Optical SATA power cable SFF R730	$13.73
Dell 0P2R3R	HBA H330 12gbps SAS SATA Mini Mono	$15.90
LSI SAS 9300-8E	8-Port 12Gb/s 12Gbps External HBA Controller Bus Card	$37.09
Netapp DS2246	Storage Expansion Array 24 Bay 2.5" SAS Trays 2x IOM6 Controllers	$149.94
2x External SAS Cable	SFF-8436 QSFP to SFF-8644 HD Hybrid SAS Cable 1.64 ft.	$41.54

Total price: $797.47 (mSATA drives and adapter cards salvaged)

Network Interfaces¶

Interface	MAC	Reserved IP	Notes
eno1	c8:1f:66:dd:6a:24	192.168.8.10	primary gigabit interface
eno2	c8:1f:66:dd:6a:25	--	secondary gigabit (unused)
eno3	c8:1f:66:dd:6a:26	--	third gigabit interface (unused)
eno4	c8:1f:66:dd:6a:27	--	fourth gigabit interface (unused)
enp132s0	24:8a:07:e3:14:80	--	10G fiber interface (unused)
enp132s0d1	24:8a:07:e3:14:81	10.250.1.6	10G fiber interface
ipmi	18:66:da:a2:a0:3b	192.168.208.4	IPMI interface

Note: 10G interface provided by Mellanox 10G PCIe card.

Storage Configuration¶

The r730xd has 24x 1TB drives and the attached Netapp shelf has an additional 24x 1TB drives connected via the LSI SAS 9300-8E card for a total of 44 TB. T

• Script to partition each device (add additional drives to the list to partition more)
  

  for device in sdar; do echo $device; parted --script /dev/$device \
      mklabel gpt \
      mkpart primary 0% 100%; done
  

• Command to create the RAID-6 array (Note: do not copy verbatim as the device names may change):
  

  $ sudo mdadm --create --verbose /dev/md0 --level=6 --raid-devices=48 /dev/sda1 /dev/sdaa1 /dev/sdab1 /dev/sdad1 /dev/sdae1 /dev/sdaf1 /dev/sdag1 /dev/sdah1 /dev/sdai1 /dev/sdaj1 /dev/sdak1 /dev/sdal1 /dev/sdam1 /dev/sdan1 /dev/sdao1 /dev/sdap1 /dev/sdaq1 /dev/sdar1 /dev/sdas1 /dev/sdat1 /dev/sdau1 /dev/sdav1 /dev/sdaw1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 /dev/sdn1 /dev/sdo1 /dev/sdp1 /dev/sdq1 /dev/sdr1 /dev/sds1 /dev/sdt1 /dev/sdu1 /dev/sdv1 /dev/sdw1 /dev/sdx1 /dev/sdy1 /dev/sdz1
  

• The create command above takes a significant amount of time to create a new array (20+ hrs). Use this command to monitory status:
  

  $ watch cat /proc/mdstat
  

• The following are miscellaneous mdadm commands for reference:
  

  $ sudo mdadm --stop /dev/md0
  $ sudo mdadm --assemble --scan
  $ sudo mdadm --manage /dev/md0 --add /dev/sdas1
  

• Use this command to store the array information to the mdadm configuration file:
  

  $ sudo su
  # mdadm --detail --scan >> /etc/mdadm/mdadm.conf
  # update-initramfs -u
  

NFS setup:¶

• Install NFS (note be sure to configure the firewall first):
  

  $ sudo apt install nfs-kernel-server
  

• Edit /etc/exports:
  

  /storage    192.168.8.0/24(rw,nohide,insecure,no_subtree_check,async)
  /storage    192.168.4.0/24(rw,nohide,insecure,no_subtree_check,async)
  
  /storage    10.250.1.0/24(rw,nohide,insecure,no_subtree_check,async)
  

• Restart NFS:
  

  $ sudo systemctl restart nfs-kernel-server
  

• Export filesystems:
  

  $ sudo exportfs
  

Firewall Configuration¶

Port	Description
22	ssh
80	http
111	SUN rpc
443	https
2049	NFS
3142	apt-cacher-ng
4045	lockd
17641	NFS connect
61993	bt

It was a bit annoying to get NFS working when the firewall was enabled but I found some good instructions and got them to work. The issue was that some of the important rpc/NFS daemons use random ports when they boot up.

• Edit /etc/default/nfs-kernel-server comment out this line:
  

  RPCMOUNTDOPTS=--manage-gids
  

  
  add this line
  

  RPCMOUNTDOPTS="--port 17641" 
  

• Edit/create /etc/modprobe.d/options.conf:
  

  options lockd nlm_udpport=4045 nlm_tcpport=4045
  

• Edit /etc/modules and add lockd on its own line:
  

  ...
  lockd
  

• Now create the firewall rules (only allow incoming access for the local network), for each port in the table above:
  

  $ sudo ufw allow from 192.168.8.0/24 to any port <PORT>
  

• Add external allow rules (i.e. up these ports to the outside world) for ssh, https, and bt :
  

  $ sudo ufw allow <PORT>
  

• Start ufw:
  

  $ sudo ufw enable
  

  
  Overview of added rules:
  

  sudo ufw allow proto tcp to 0.0.0.0/0 port 22
  sudo ufw allow proto tcp to 0.0.0.0/0 port 443
  sudo ufw allow proto tcp to 0.0.0.0/0 port 61993
  
  sudo ufw allow from 192.168.4.0/24 to any port 111
  sudo ufw allow from 192.168.4.0/24 to any port 2049
  sudo ufw allow from 192.168.4.0/24 to any port 4045
  sudo ufw allow from 192.168.4.0/24 to any port 17641
  
  sudo ufw allow from 192.168.8.0/24 to any port 111
  sudo ufw allow from 192.168.8.0/24 to any port 2049
  sudo ufw allow from 192.168.8.0/24 to any port 4045
  sudo ufw allow from 192.168.8.0/24 to any port 17641
  
  sudo ufw allow from 10.250.1.0/24 to any port 111
  sudo ufw allow from 10.250.1.0/24 to any port 2049
  sudo ufw allow from 10.250.1.0/24 to any port 4045
  sudo ufw allow from 10.250.1.0/24 to any port 17641