Project

General

Profile

Actions
History

Wiki »

brutus Server Setup

History

Who needs "the cloud", I've got gigabit fiber running directly to my house!
-- Me

I've been running a local file server for 16ish years.

Date Note
7/28/2025 Upgraded to 256GB (+128GB) RAM
9/11/2024 Swtiched to Dynu for dynamic DNS and used Let's Encrypt for a proper SSL/TLS cert.
9/8/2024 Retired phalanx and replaced it with brutus (hardware details in the section below).
5/27/2022 Wipe and install Ubuntu 22.04
7/8/2019 Change Dynamic DNS provider to http://freedns.afraid.org.
1/9/2019 Added 10G Ethernet interface.
1/4/2019 Replaced mobo/proc/RAM with parts salvaged from work. Proc i7-4790 (Haswell); RAM 32GB DDR3L.
6/2/2018 Upgraded boot drive to 500 GB SSD. Also installed Ubuntu 18.04 LTS.
1/16/2017 Upgraded server added 2 x 3 TB additional HDDs (via PCIe SATA expansion card), new power supply, and new (slightly used) case. After adding the new disks to the RAID and growing the file system, total usable storage on the array is now 11.18 TB.
11/18/2016 Updated (wipe and reinstall) to Ubuntu Server 16.04.1 LTS
9/2015 I recently decided to pay for dynamic DNS service from Dyn. The cost was $30 (after 25% coupon) for a year w/ 30 hostnames. Server is now reachable at https://baranovich.homelinux.org:3268
8/2014 Both of the my original 3TB HDDs started acting funny and died within a week of each other after about 1.8 years of continuous use. Fortunately I reacted immediately and I was able to back up all of my data onto other disks. Although I regularly back up all of my critical data I decided that I really don't want to have to deal with losing any of my data. I promptly purchased another HDD and a HW RAID card (an LSI MegaRAID SAS 9240-8i) and set out to build a RAID 5 array. I quickly ran into problems with the RAID card. I was only able to get the computer to boot with it installed once. During that boot I upgraded the card's firmware with the hope that it would fix all my issues, but it did not and I was never able to fully boot with the card installed again. After a little googling I discovered someone in my same position who claimed to have called LSI tech support and was told that LSI cards are "not compatible with newer motherboards". The guy didn't elaborate as to what that meant but I assume it means LSI cards don't work in UEFI motherboards. I was disappointed but I started reading about software raid (device mapper raid) and decided that it should work perfectly for my little fileserver. I also decided that btrfs is not really ready for prime time and I should use some other file system.
??/2008 Created first version of phalanx with Dell desktop that was Kab's college computer.

Hardware Details

  • Dell r730xd server 2U
    • 2x Intel E5-2680v4 CPUs
    • 128 256 GB DDR4 2400
    • 24x 1 TB mSATA PM-851
    • Dell HBA H330
    • LSI SAS 9300-8E 8-Port 12Gb/s 12Gbps External HBA Controller Bus Card
  • Netapp DS2246 Storage Expansion Array 24 Bay 2.5" SAS Trays 2x IOM6 Controllers
    • 24x 1 TB mSATA PM-851

Total raw storage capacity: 44 TB
Total usable storage capacity (RAID-6): 42.84 TB

Price Breakdown

Components purchased from e-Bay.

Component Description Price
Dell r730xd server 2x Intel E5-2680v4 CPUs; 128 GB DDR4 $413.39
24x 2.5" 8FKXC/08FKXC SAS SATA Drive Caddy For Dell PowerEdge $88.79
Dell TRJ5G Optical SATA power cable SFF R730 $13.73
Dell 0P2R3R HBA H330 12gbps SAS SATA Mini Mono $15.90
LSI SAS 9300-8E 8-Port 12Gb/s 12Gbps External HBA Controller Bus Card $37.09
Netapp DS2246 Storage Expansion Array 24 Bay 2.5" SAS Trays 2x IOM6 Controllers $149.94
2x External SAS Cable SFF-8436 QSFP to SFF-8644 HD Hybrid SAS Cable 1.64 ft. $41.54
128 GB DDR4-2400 RAM $117.78

Total price: $797.47 (mSATA drives and adapter cards salvaged)

Network Interfaces

Interface MAC Reserved IP Notes
eno1 c8:1f:66:dd:6a:24 192.168.8.10 primary gigabit interface
eno2 c8:1f:66:dd:6a:25 -- secondary gigabit (unused)
eno3 c8:1f:66:dd:6a:26 -- third gigabit interface (unused)
eno4 c8:1f:66:dd:6a:27 -- fourth gigabit interface (unused)
enp132s0 24:8a:07:e3:14:80 -- 10G fiber interface (unused)
enp132s0d1 24:8a:07:e3:14:81 10.250.1.6 10G fiber interface
ipmi 18:66:da:a2:a0:3b 192.168.208.4 IPMI interface

Note: 10G interface provided by Mellanox 10G PCIe card.

Storage Configuration

The r730xd has 24x 1TB drives and the attached Netapp shelf has an additional 24x 1TB drives connected via the LSI SAS 9300-8E card for a total of 44 TB. T

  • Script to partition each device (add additional drives to the list to partition more)
    for device in sdar; do echo $device; parted --script /dev/$device \
    mklabel gpt \
    mkpart primary 0% 100%; done
  • Command to create the RAID-6 array (Note: do not copy verbatim as the device names may change):
    $ sudo mdadm --create --verbose /dev/md0 --level=6 --raid-devices=48 /dev/sda1 /dev/sdaa1 /dev/sdab1 /dev/sdad1 /dev/sdae1 /dev/sdaf1 /dev/sdag1 /dev/sdah1 /dev/sdai1 /dev/sdaj1 /dev/sdak1 /dev/sdal1 /dev/sdam1 /dev/sdan1 /dev/sdao1 /dev/sdap1 /dev/sdaq1 /dev/sdar1 /dev/sdas1 /dev/sdat1 /dev/sdau1 /dev/sdav1 /dev/sdaw1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 /dev/sdn1 /dev/sdo1 /dev/sdp1 /dev/sdq1 /dev/sdr1 /dev/sds1 /dev/sdt1 /dev/sdu1 /dev/sdv1 /dev/sdw1 /dev/sdx1 /dev/sdy1 /dev/sdz1
  • The create command above takes a significant amount of time to create a new array (20+ hrs). Use this command to monitory status:
    $ watch cat /proc/mdstat
  • The following are miscellaneous mdadm commands for reference:
    $ sudo mdadm --stop /dev/md0
$ sudo mdadm --assemble --scan
$ sudo mdadm --manage /dev/md0 --add /dev/sdas1
  • Use this command to store the array information to the mdadm configuration file:
    $ sudo su
# mdadm --detail --scan >> /etc/mdadm/mdadm.conf
# update-initramfs -u

NFS setup:

  • Install NFS (note be sure to configure the firewall first):
    $ sudo apt install nfs-kernel-server
  • Edit /etc/exports:
    /storage    192.168.8.0/24(rw,nohide,insecure,no_subtree_check,async)
/storage    192.168.4.0/24(rw,nohide,insecure,no_subtree_check,async)

/storage    10.250.1.0/24(rw,nohide,insecure,no_subtree_check,async)
  • Restart NFS:
    $ sudo systemctl restart nfs-kernel-server
  • Export filesystems:
    $ sudo exportfs

Firewall Configuration

Port Description
22 ssh
80 http
111 SUN rpc
443 https
2049 NFS
3142 apt-cacher-ng
4045 lockd
17641 NFS connect
61993 bt

It was a bit annoying to get NFS working when the firewall was enabled but I found some good instructions and got them to work. The issue was that some of the important rpc/NFS daemons use random ports when they boot up.

  • Edit /etc/default/nfs-kernel-server comment out this line:
    RPCMOUNTDOPTS=--manage-gids

    add this line
    RPCMOUNTDOPTS="--port 17641"
  • Edit/create /etc/modprobe.d/options.conf:
    options lockd nlm_udpport=4045 nlm_tcpport=4045
  • Edit /etc/modules and add lockd on its own line:
    ...
lockd
  • Now create the firewall rules (only allow incoming access for the local network), for each port in the table above:
    $ sudo ufw allow from 192.168.8.0/24 to any port <PORT>
  • Add external allow rules (i.e. up these ports to the outside world) for ssh, https, and bt :
    $ sudo ufw allow <PORT>
  • Start ufw:
    $ sudo ufw enable

    Overview of added rules:
    sudo ufw allow proto tcp to 0.0.0.0/0 port 22
sudo ufw allow proto tcp to 0.0.0.0/0 port 443
sudo ufw allow proto tcp to 0.0.0.0/0 port 61993

sudo ufw allow from 192.168.4.0/24 to any port 111
sudo ufw allow from 192.168.4.0/24 to any port 2049
sudo ufw allow from 192.168.4.0/24 to any port 4045
sudo ufw allow from 192.168.4.0/24 to any port 17641

sudo ufw allow from 192.168.8.0/24 to any port 111
sudo ufw allow from 192.168.8.0/24 to any port 2049
sudo ufw allow from 192.168.8.0/24 to any port 4045
sudo ufw allow from 192.168.8.0/24 to any port 17641

sudo ufw allow from 10.250.1.0/24 to any port 111
sudo ufw allow from 10.250.1.0/24 to any port 2049
sudo ufw allow from 10.250.1.0/24 to any port 4045
sudo ufw allow from 10.250.1.0/24 to any port 17641

Apache

The Apache config (see repo) is the really important part here. Below is a section on each application that's running on/from Apache. I am using https with a permanent redirect from port 80 along with a Let's Encrypt provided SSL/TLS certificate. Most of the sections below are password protected using mod_auth_digest. The appropriate digest file is created and managed with the htdigest tool. NOTE: make sure the password digest file (e.g. /home/dsorber/web_users) is owned by www-data:www-data and permissions are set to 600.

  • Install apache2 itself:
    $ sudo apt install apache2
  • Enable apache modules:
    $ sudo a2enmod ssl proxy proxy_http auth_digest
  • Copy backed up apache configs to their location:
    $ sudo cp apache_configs/* /etc/apache2/sites-available/
  • Turn on main site (also disable default apache2 site):
    $ sudo a2ensite main

Dynamic DNS

Free Dynamic DNS is provided by Dynu

  • Install ddclient
    $ sudo apt install ddclient
  • Edit /etc/ddclient:
    daemon=60                                                    # Check every 60 seconds.
syslog=yes                                                   # Log update msgs to syslog.
mail=root                                                    # Mail all msgs to root.
mail-failure=root                                            # Mail failed update msgs to root.
pid=/var/run/ddclient.pid                                    # Record PID in file.
use=web, web=checkip.dynu.com/, web-skip='IP Address'        # Get ip from server.
server=api.dynu.com                                          # IP update server.
protocol=dyndns2
login=baranovich                                             # Your username
password=PASSWORD_GOES_HERE                                  # Your password or MD5/SHA256 hash of password
brutus.gleeze.com                                            # Update IP address for alias of your domain name
  • Restart ddclient:
    $ sudo systemctl restart ddclient

Let's Encrypt Setup

  • Install cerbot in a VirtualEnv and create a link
    $ sudo apt install python3.12-venv
$ sudo python3 -m venv /opt/certbot
$ sudo /opt/certbot/bin/pip install --upgrade pip
$ sudo /opt/certbot/bin/pip install certbot
$ sudo /opt/certbot/bin/pip install certbot-dns-dynu
$ sudo ln -s /opt/certbot/bin/certbot /usr/bin/certbot
  • Create dynu-credentials.ini see below for instructions for getting token
    dns_dynu_auth_token = [REDACTED]
  • Run certbot
    $ sudo certbot --authenticator dns-dynu --dns-dynu-credentials ./dynu-credentials.ini certonly
  • Update Apache configuration main.conf
  • Add cronjob for automated renewal:
    $ echo "0 0,12 * * * root /opt/certbot/bin/python -c 'import random; import time; time.sleep(random.random() * 3600)' && sudo certbot renew -q" | sudo tee -a /etc/crontab > /dev/null
Resources

Transmission

I use Transmission to download Linux .isos ( wink ) and give back to the community by sharing them using my own bandwidth. Transmission with its webui runs as a separate daemon so apache is configured to act as reverse proxy.

  • Install transmission daemon:
    $ sudo apt install transmission-daemon
  • Copy over configuration:
    $ sudo systemctl stop transmission-daemon.service
$ sudo cp settings.json /etc/transmission-daemon
$ sudo systemctl start transmission-daemon.service
  • Make sure the user digest file is in place (NOTE: be sure to change use directory permissions to 755 or Apache will not be able to read the web_users file):
    $ cd /home/dsorber
$ cp web_users .
$ cd /home
$ sudo chmod 755 dsorber
  • Enable site:
    $ sudo a2ensite transmission

Redmine + Git

Unfortunately Trac completely stagnated and never completed moving to Python 3. Therefore I switched to Redmine + Git which (at least as of now) is still maintained. Note the following uses PostgresQL for the backend database (which is also used for nextcloud).

  1. Install prereqs:
     $ sudo apt install apache2 pbzip2 zip unzip postgresql libpq-dev build-essential ruby ruby-dev
  2. Download Redmine:
    $ cd /srv/
$ sudo wget https://www.redmine.org/releases/redmine-5.1.4.tar.gz
$ sudo tar xf redmine-5.1.4.tar.gz
$ sudo mv redmine-5.1.4 redmine
$ sudo chown -R root:root redmine
  3. Setup postgres database for Redmine:
    $ sudo -u postgres psql postgres
> CREATE ROLE redmine LOGIN ENCRYPTED PASSWORD 'password_goes_here' NOINHERIT VALID UNTIL 'infinity';
> CREATE DATABASE redmine WITH ENCODING='UTF8' OWNER=redmine;
Ctrl+D
  4. Create database config
    $ cd /srv/redmine/config
$ sudo cp database.yml.example database.yml
    • Edit database.yml:
      production:
  adapter: postgresql
  database: redmine
  host: localhost
  username: redmine
  password: "PASSWORD_GOES_HERE" 

# PostgreSQL configuration example
#production:
#  adapter: postgresql
#  database: redmine
#  host: localhost
#  username: postgres
#  password: "postgres" 

# SQLite3 configuration example
#production:
#  adapter: sqlite3
#  database: db/redmine.sqlite3

# SQL Server configuration example
#production:
#  adapter: sqlserver
#  database: redmine
#  host: localhost
#  username: jenkins
#  password: jenkins
  5. Install bundle and dependencies:
    $ sudo gem install bundler
$ cd /srv/redmine
$ sudo bundle config set --local without 'development test'
$ sudo bundle install
  6. Configure Redmine database:
    $ sudo rake generate_secret_token
$ sudo RAILS_ENV=production rake db:migrate
$ sudo RAILS_ENV=production rake redmine:load_default_data
  7. Configure permissions:
    $ sudo chown -R www-data:www-data files log tmp public/plugin_assets
$ sudo chmod -R 755 files log tmp public/plugin_assets
  8. Setup webserver:
    $ sudo chown www-data:www-data /srv/redmine/config/environment.rb
$ sudo ln -s /srv/redmine/public /var/www/redmine
  9. Install Passenger:
    $ sudo apt install libcurl4-openssl-dev libssl-dev apache2-dev libapr1-dev libaprutil1-dev zlib1g-dev
    • Create file Gemfile.local:
      gem 'passenger'
    • Continue with install:
      $ sudo bundle install
$ sudo passenger-install-apache2-module (follow instructions)
$ sudo vim /etc/apache2/mods-available/passenger.load
-------------------------------------------------------------------------------
LoadModule passenger_module /var/lib/gems/3.0.0/gems/passenger-6.0.14/buildout/apache2/mod_passenger.so
-------------------------------------------------------------------------------

$ sudo vim /etc/apache2/mods-available/passenger.conf
-------------------------------------------------------------------------------
  <IfModule mod_passenger.c>
     PassengerRoot /var/lib/gems/3.0.0/gems/passenger-6.0.14
     PassengerDefaultRuby /usr/bin/ruby3.0
   </IfModule>
-------------------------------------------------------------------------------

$ sudo a2enmod passenger
$ sudo systemctl restart apache2
  10. Setup Apache config. Note the Redmine section is now included in main.conf and should be commented out until this step is reached.
  11. Install theme(s):
    $ cd /srv/redmine/public/themes
$ git clone https://github.com/koppen/redmine-pepper-theme.git
$ git clone https://github.com/makotokw/redmine-theme-gitmike

Redmine Backup

There are two items that need to be backed up, the database and any attached files:

  • Backup:
    $ /usr/bin/pg_dump -U redmine -h localhost -Fc --file=redmine.sqlc redmine

$ cd /srv/redmine/
$ tar cjf redmine_files.tar.bz2 files
  • Restore:
    $ pg_restore -U redmine -h localhost -d redmine redmine.sqlc

$ tar xf redmine_files.tar.bz2
$ sudo cp -r files/ /srv/redmine/files

Git Integration

  1. Install prereqs:
    $ sudo apt install libapache2-mod-perl2 libdbi-perl libdbd-pg-perl
  2. Configure:
    $ sudo ln -s /srv/redmine/extra/svn/Redmine.pm /usr/lib/x86_64-linux-gnu/perl5/5.34/Apache2/Redmine.pm
$ sudo a2enmod perl dav dav_fs
  3. Setup git repo. Note if using an existing repo then just copy it into path: /srv/git
    $ sudo mkdir -p /srv/git
$ cd /srv
$ sudo chown -R dsorber:dsorber git
(optional used to test below)
$ cd git 
$ mkdir test; cd test
 $ git init --bare
  4. Setup grack:
    $ cd /var/www
$ sudo git clone https://github.com/grackorg/grack.git
$ sudo vim grack/config.ru
-------------------------------------------------------------------------------
require 'grack/app'
require 'grack/git_adapter'

config = {
  :root => '/srv/git',
  :allow_push => true,
  :allow_pull => true,
  :git_adapter_factory => ->{ Grack::GitAdapter.new }
}

run Grack::App.new(config)

-------------------------------------------------------------------------------

$ sudo chown -R www-data:www-data grack
$ cd /var/www/grack
$ sudo mkdir public
$ sudo mkdir tmp
$ sudo chown -R www-data:www-data .
$ sudo bundle install
  5. Configure apache. Note the git section is now included in main.conf and should be commented out until this step is reached.
    $ sudo a2ensite git 
$ sudo systemctl restart apache2

Nextcloud

Nextcloud is a self-hosted "cloud". It allows easy file viewing and sharing as well as CardDav and CalDav servers:

  1. Install prereqs:
    $ sudo apt install apache2 libapache2-mod-php php-gd php-pgsql php-curl php-mbstring php-intl php-gmp php-bcmath php-xml php-imagick php-zip
  2. Download and move into place:
    $ wget https://download.nextcloud.com/server/releases/latest.tar.bz2
$ tar xf latest.tar.bz2
$ sudo mv nextcloud /var/www/nextcloud
$ cd /var/www
$ sudo chown -R www-data:www-data nextcloud
  3. Config database (use postgresql; same as redmine):
    $ sudo -u postgres psql postgres
> CREATE ROLE nextcloud LOGIN ENCRYPTED PASSWORD 'password_goes_here' NOINHERIT VALID UNTIL 'infinity';
> CREATE DATABASE nextcloud WITH ENCODING='UTF8' OWNER=nextcloud;
Ctrl+D
  4. Setup Apache config. Note the Nextcloud section is now included in main.conf and should be commented out until this step is reached.
    $ sudo a2enmod rewrite headers env dir mime
$ sudo vim /etc/apache2/envvars
-------------------------------------------------------------------------------
## Uncomment the following line to use the system default locale instead:

. /etc/default/local here
-------------------------------------------------------------------------------

$ sudo systemctl restart apache2
  5. Complete configuration by navigating to https://brutus.gleeze.com/nextcloud in a web browser and following the instructions.

Other configuration

  • To setup local storage: go to Apps > Disabled Apps > External Storage (click Enable)
  • To setup pretty URLS: * https://docs.nextcloud.com/server/13/admin_manual/installation/source_installation.html#pretty-urls
    • Edit /var/www/nextcloud/config/config.php:
        'htaccess.RewriteBase' => '/nextcloud',
  'default_phone_region' => 'US',
  'maintenance_window_start' => 1,
    • Edit /etc/php/8.3/apache2/php.ini change line to:
      memory_limit = 1024M
opcache.interned_strings_buffer=64
opcache.revalidate_freq=60
    • Add missing indices:
      $ cd /var/www/nextcloud
$ sudo -u www-data php occ db:add-missing-indice
    • Restart apache:
      $ sudo systemctl restart apache2

Misc

  • Setup the display script: * Edit ~/.profile and add ./repo/software/misc/display.py
  • Setup checker script: * 
    $ sudo su
# crontab -e

    add this line:
    0 * * * *       /home/dsorber/repo/software/misc/checker.py

Updated by 8 months ago · 13 revisions