Intel NUC running ESXI

Ended up moving my server over to an Intel NUC D54250WYK Haswell i5 with attached storage.  Ah, the advantages of running a VM.  Intel finally released this low-power desktop system and it seems to work great with Esxi 5.5, with the addition of a few drivers.

The problem is described here. However, in fact, the driver the author provided for the e1001 did not work for my system.  I used a combination of the 5.5 iso (VMware-VMvisor-Installer-5.5.0-1331820.x86_64.iso), ESXI-customizer 2.7.1, and two drivers, the e1000 (net-e1000e-2.3.2.x86_64.vib) from here and sata-xhci (sata-xahci-1.5-1.x86_64.vib) from here.  I've provided a mirror for the drivers and the customerizer on dropbox here.
The ESXI-customizer takes care of most of the nitty gritty, so just follow the instructions there for adding the drivers, one at a time.

VT-d and PCI passthrough seems to work just fine with this hardware.

IRC Web Client

For the web interface, we will install qwebirc and proxy it through Nginx on Ubuntu 12.04 Precise Pangolin.  The instructions for qwebirc are kind of in different places, so hopefully this will help someone.

In my case JDK is already installed, so the dependencies I needed where

sudo apt-get install python python-twisted python-twisted-bin \
   python-twisted-core python-twisted-runner python-twisted-names \ 
   python-twisted-mail python-twisted-words python-twisted-web \
   python-zope.interface python-openss mercurial

Download the source somewhere

hg clone http://hg.qwebirc.org/qwebirc qwebirc

I decided to switch to their stable branch

hg up -C stable

Ok, in my case, I needed to apply two patches for SSL and server authentication.  For the authentication I applied the patch from here, direct link to the patch here.  For the SSL support I applied the patch from anacart's post in this thread, direct link to the patch here.

To apply the patch cd to the source root and do

patch -p1 < patch.diff

Once your done, put the qwebirc folder somewhere permanent, like /usr/local/qwebirc, or /usr/share/qwebirc, and make a copy of the config file.

cp config.py.example config.py

Edit config.py. Change IRCPORT and SSL port to match the client port of your IRC server.

Set IDENT to a valid user on your LDAP domain.  I created an account called "webirc" in ldap-account-manager.

IDENT = "webirc"

Set the NETWORK to the IRC network name, specified in the inspircd.conf

NETWORK_NAME = "IRCNet"

I wasn't sure what to see the URLs to, but here's how mine is set. Set REALNAME to the server address.

REALNAME = "https://www.domain.com/webirc"

 Set BASE_URL to the local address, i don't think this is right, needs checking.

BASE_URL = "http://localhost:9090"

For the Nginx proxy, set the following.

FORWARDED_FOR_HEADER="x-forwarded-for"
FORWARDED_FOR_IPS=["127.0.0.1"]

Finally compile qwebirc.

python compile.py

And test it

python run.py 

You should be able to browse to http://localhost:9090/

Lastly, create a file to launch qwebirc as a service.  If qwebirc crashes, this script will not restart the process, it needs some tweaking. Edit /etc/init/qwebirc.conf

# qwebirc - qwebirc job file
start on runlevel [2345]stop on runlevel [016]
chdir /usr/local/qwebirc
export fork
exec /usr/local/qwebirc/run.py

If all goes well, it should start on reboot, or by running

sudo service qwebirc start

Now to tweak Nginx.  I just had to add the following to /etc/nginx/sites-enabled/default

location ^~ /webirc/ {
        proxy_set_header  Host             $host;
        proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header  X-Forwarded-Proto https;
        proxy_pass  http://127.0.0.1:9090/;
}

Restart nginx

sudo service nginx restart

And browse to https://www.domain.com/webirc 

IRC Server

Decided to setup a little IRC server using Inspircd.  The plan was as follows. Use LDAP authentication for users, SSL encryption using my domain certs, a web-based UI proxied through NGINX https and lastly, federated with a friend's IRC server. Ubuntu 12.04 comes with an old version of Inspircd (1.1.2?), doesn't work well with federation and doesn't come with the LDAP module.  I downloaded the latest version of Inspircd that comes with 12.10 Ubuntu from here.

The above link also shows any dependencies you may need to install separately.  I was missing a few, solved with...

sudo apt-get install libtre5 libpq5 libmysqlclient18

then installed Inspircd

sudo dpkg -i inspircd_2.0.5-1_amd64.deb

First you need to edit /etc/default/inspircd and change the '0' to '1'

The main config file to edit is /etc/inspircd/inspircd.conf You'll want to configure this to setup some basic info, like change the <bind> tag to listen on an ip other than "127.0.0.1". "" will default to all network interfaces, and then set the port to listen to.

At this point, you can run the Inspircd daemon

sudo service inspircd start

Next will secure the chat client port to use the SSL cert for the server. I store the SSL certs with my nginx server in /etc/nginx/certs. 

First you need to tell Inspircd to load the gnutls module and point to your certs, by editing /etc/inspircd/inspircd.conf and adding:

<module name="m_ssl_gnutls.so">
<gnutls certfile="/etc/nginx/certs/server.crt" keyfile="/etc/nginx/certs/server.key">

If you want, you can create a self-signed cert, and use that, but clients will need to be told to ignore invalid certs.

Next, change your client's bind tag to something like:

<bind address="" port="5309" type="clients" ssl="gnutls">

To add LDAP authentication, you need to load the ldapauth module and point to your ldap server, by editing /etc/inspircd/inspircd.conf and adding:

<module name="m_ldapauth.so">
<ldapauth baserdn="ou=People,dc=domain,dc=com"
          attribute="uid"
          server="ldap://localhost"
          allowpattern="Guest*"
          killreason="Access denied"
          searchscope="subtree"
          binddn=""
          bindauth=""
          verbose="yes"
          userfield="yes">

To connect this server to another server, you need to <bind> a port as type server,

<bind address="" port="9799" type="servers">

setup a <link> section to define the server connection.  The same thing needs to be setup on the other server to be connected.

<link name="irc.otherdomain.com"
      ipaddr="irc.otherdomain.com"
      port="9799"
      sendpass="secret"
      recvpass="secret">

Lastly, one of the two servers can be set to <autoconnect> to avoid manually maintaining the connection.

<autoconnect period="60" server="irc.otherdomain.com">

Part 2 of this blog entry will setup the web interface

Self Service Password

I decided to allow users on the server to change their passwords, when they want, through a web based tool. I chose LTB's Self Service Password. A simple php tool with lots of neat features like SMS reset, security questions, etc.  I only plan to enable the simple form to reset the password. To install, download the latest .deb file, (0.8 in my case). Next install the dependencies, and restart php5.

sudo apt-get install apache2 php5 php5-ldap php5-mcrypt
sudo service php5-fpm restart

Then the .deb

sudo dpkg -i self-service-password_0.8-1_all.deb

You will need to modify php config file at /usr/share/self-service-password/conf/config.inc.php and make some changes to LDAP.

In my case, the server runs on the localhost.

$ldap_url = "ldap://localhost";

ldap_binddn and ldap_bindpw are made blank ("") to not use admin credentials.

$ldap_binddn = "";
$ldap_bindpw = "";

ldap_base is set to your domain.

$ldap_base = "dc=domain,dc=com";

I'm using simple posix schema for users.

$ldap_filter = "(&(objectClass=posixAccount)($ldap_login_attribute={login}))";   

Next up, modify your nginx config file at /etc/nginx/sites-enabled/default,and add the following sections.

#Self Service Password Section
location /self-service-password {
        alias /usr/share/self-service-password;
        index index.html index.php;
}

location ~ ^/self-service-password/.*\.php$ {
        root /usr/share;
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $request_filename;
        include /etc/nginx/fastcgi_params;
}

Restart nginx and browse to https://www.domain.com/self-service-password

sudo service nginx restart 

Add Authentication to Nginx

The plan is to expose certain web apps behind Basic HTTP Authentication.  This is why the web server is only available via HTTPS, we do not want LDAP user/passwords going over plaintext.  The way we have setup LDAP and PAM earlier, it is very easy to secure subdomains using it. Note: Basic HTTP Authentication has uses no session cookies or persistence, so the user remains logged in until they close their browser. Be warned!

Make sure that you have nginx-extras installed, and not nginx.  Extras includes the PAM module.

sudo apt-get install nginx-extras

The following is taken from this readme.  
To protect everything under /secure you will add the following to the nginx.conf file: This is secure enough for many purposes.

location /secure {
    auth_pam              "Secure Zone";
    auth_pam_service_name "nginx";
}

Note that the module runs as the web server user, so the PAM modules used must be able to authenticate the users without being root; that means that if you want to use the pam_unix.so module to authenticate users you need to let the web server user to read the /etc/shadow file if that does not scare you (on Debian like systems you can add the www-data user to the shadow group).

As an example, to authenticate users against an LDAP server (using the pam_ldap.so module) you will use an /etc/pam.d/nginx like the following:

auth    required pam_ldap.so
account required pam_ldap.so

If you also want to limit the users from LDAP that can authenticate you can use the pam_listfile.so module; to limit who can access resources under /restricted add the following to the nginx.conf file:

location /restricted {
    auth_pam              "Restricted Zone";
    auth_pam_service_name "nginx_restricted";
}

Use the following /etc/pam.d/nginx_restricted file:

auth    required pam_listfile.so onerr=fail item=user \
                 sense=allow file=/etc/nginx/restricted_users
auth    required pam_ldap.so
account required pam_ldap.so

And add the users allowed to authenticate to the /etc/nginx/restricted_users (remember that the web server user has to be able to read this file).

Build your ownCloud

ownCloud is a way to bring your own cloud storage to the internets.  You may want to use this to control your own data, use open standards provided by ownCloud, or just save yourself the monthly subscription costs of the commercial options. The only limitation on storage size is the size of your connected storage, which can actually include other cloud storage services such as Dropbox and Google Drive, in addition to external storage.

Installation is fairly easy with Ubuntu as a repository with packages is available. The following directories are taken from the installation page on ownCloud. Run the following as root

echo 'deb http://download.opensuse.org/repositories/isv:ownCloud:community/xUbuntu_12.04/ /' >> /etc/apt/sources.list.d/owncloud.list
apt-get update
apt-get install owncloud

If you want to add the key to apt-get to avoid a warning

wget http://download.opensuse.org/repositories/isv:ownCloud:community/xUbuntu_12.04/Release.key
apt-get add - < Release.key

Once installed, you need to add ownCloud to Nginx.  You will presumably already setup php5-fpm during the instructions in my LDAP server post.  With ownCloud 5, some slightly more complicated Nginx rules are needed. Again, we are putting ownCloud in a subdomain on the webserver for clean separation of server services. Note: make sure your fastcgi_pass matches the mechanism you are using for FastCGI, either unix socket or tcp socket.  The following was taken from this post. 

#owncloud settings
#Some rewrite rules, more to come later
rewrite ^/owncloud/caldav((/|$).*)$ /owncloud/remote.php/caldav$1 last;
rewrite ^/owncloud/carddav((/|$).*)$ /owncloud/remote.php/carddav$1 last;
rewrite ^/owncloud/webdav((/|$).*)$ /owncloud/remote.php/webdav$1 last;

location ~ ^/owncloud/(data|config|\.ht|db_structure.xml|README) {
    deny all;
}
# Configure the root location with proper rewrite rule
location /owncloud/ {
    rewrite ^/owncloud/.well-known/host-meta /public.php?service=host-meta last;
    rewrite ^/owncloud/.well-known/host-meta.json /public.php?service=host-meta-json last;
    rewrite ^/owncloud/.well-known/carddav /remote.php/carddav/ redirect;
    rewrite ^/owncloud/.well-known/caldav /remote.php/caldav/ redirect;
    rewrite ^/owncloud/apps/calendar/caldav.php /remote.php/caldav/ last;
    rewrite ^/owncloud/apps/contacts/carddav.php /remote.php/carddav/ last;
    rewrite ^/owncloud/apps/([^/]*)/(.*\.(css|php))$ /index.php?app=$1&getfile=$2 last;
    rewrite ^(/owncloud/core/doc[^\/]+/)$ $1/index.html;
    try_files $uri $uri/ index.php;
}
# Configure PHP-FPM stuff
location ~ ^(?<script_name>.+?\.php)(?<path_info>/.*)?$ {
    try_files $script_name = 404;
    fastcgi_pass unix:/var/run/php5-fpm.sock;
    fastcgi_param PATH_INFO $path_info;
    fastcgi_param HTTPS on;
    # This one is a little bit tricky, you need to pass all parameters in a single line, separating them with newline (\n)
    fastcgi_param PHP_VALUE "upload_max_filesize = 1024M \n post_max_size = 1024M"; # This finishes the max upload size settings
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; # On some systems OC will work without this setting, but it doesn't hurt to leave it here
    include /etc/nginx/fastcgi_params;
}

Make sure to restart nginx as usual, sudo service nginx restart. You should now be able to create a default admin account at your domain.com/owncloud

Next step is to link ownCloud to your Ldap server for authentication. Login with the admin account, click the settings button, and go to "Apps".  Enable the app for "Ldap User and Group Backend". 

Click the settings button, and go to "Admin".  Under the LDAP section, set your LDAP host, your domain, and the user and group attributes. Test the configuration and save.

Again, if you are using ownCloud 4.5 everything should work out of the box as is, and users can login and share files with group members.  In my case, all web users are a member of the group 'webuser'.  However, ownCloud 5.0 requires some additional configuration, or the users are not associated with their groups.  The solution is to add the memberUid attribute to the associated group in ldap-account-manager, manually add the users to this group, then tell ownCloud to use this attribute.

Login to ldap-account-manager and click on "Tree View".  From here, select the group, and click "Add New Attribute".  Select "memberUid".  Add the name of at least one user.  The new attribute should be visible in the group in tree view.  From here, you can manually add members by clicking "Modify Group Members" under memberUid.  You can add the users in a batch, instead of manually typing them out.

Log back in to ownCloud as admin.  Click on the settings button, and go back into "Admin".  In the LDAP section, select the "Advanced" tab. Under "Directory Settings" set "Group-member association" to "memberUid".  Save.  You may need to remove the users so that ownCloud can repopulate the list with the new group association.

Setting up LDAP Server

LDAP Server Setup

Lightweight Directory Access Protocol or LDAP, is a high-level application protocol for managing directory services in a hierarchical manner.  It's most common use is to manage domain related information, such as an email directory or user information.  In my case, I will use the Unix-related structures for managing users and their system access to my services.  The same user name and login will be used for Subsonic, Owncloud, COPS (e-book server), SSH-SFTP logins, etc.  

To start installing OpenLDAPServer, you can use 2 guides over at Ubuntu, here and here.  The second link actual corrects a few things in the guide, but most of it is unnecessary for an initial LDAP server.  

First off, install the packages.

sudo apt-get install slapd ldap-utils

In my case, my host is already joined to a domain, so I didn't need the next step, but just to make sure, reconfigure slapd to add the ldap domain and reset the password.

sudo dpkg-reconfigure slapd

That's pretty much all you need to get running.  The latest builds of Ubuntu handle the inclusion of various basic schemas, but to verify your ldap is up and running, run the following.

sudo ldapsearch -Y EXTERNAL -H ldapi:/// -b cn=config

Initially this command should list 10-15 entries and is a good first check.

User Configuration

A great utility for managing users and settings includes a web app called LAM, or ldap-account-manager.  There is another great write-up regarding using LAM with Nginx here. 

Install the necessary packages 

sudo apt-get install php5-fpm php5 php5-ldap php-apc php5-gd php-fpdf ldap-account-manager

Normally php5-fpm is configured listening on 127.0.0.1 port 9000.  We're going to change this to a Unix socket, just to clean up the ports a bit and potentially increases performance under load. In general it won't help much, but theoretically removes some of the TCP overhead.

sudo vi /etc/php5/fpm/pool.d/www.conf

Look for 

listen = 127.0.0.1:9000

Change to

listen = /var/run/php5-fpm.sock

Restart the service

sudo service php5-fpm restart

Add the following section to /etc/nginx/sites-enabled/default to create a sub-domain for the account manager, and will point it to the main launch page.

location /ldap-account-manager {
        alias /usr/share/ldap-account-manager;
        index index.html index.php;
}

Add the following section to /etc/nginx/sites-enabled/default to point Nginx to the LAM directory, the php Unix socket, and tweak a couple of fastcgi parameters.

location ~ ^/ldap-account-manager/.*\.php$ {
        root /usr/share;
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $request_filename;
        include /etc/nginx/fastcgi_params;
}

Restart Nginx

sudo service nginx restart

You should be able to now browse to LAM https://www.domain.com/ldap-account-manager. At this point, I refer you to the ducky-pond.com post for how to initially setup LAM. 

Client Access

First, configure your client config for ldap client apps.  

sudo vi /etc/ldap.conf

Make sure the domain is properly specified

base dc=domain,dc=com

and the uri is correct

uri ldap://127.0.0.1:389

I refer you to this thread for the instructions I used to setup local and ssh logins for your users.  This will automatically create their home directories if they do not exist. There is a correction by a later contributer, which I have included in my quick setup below

First install the packages

sudo apt-get install ldap-utils libpam-ldap libnss-ldap nslcd

Next edit /etc/nsswitch.conf and change the lines for passwd, group, and shadow

passwd: compat ldap
group : compat ldap
shadow: compat ldap

Edit /etc/pam.d/lightdm and add 

session required pam_mkhomedir.so skel=/etc/skel umask=0022

Edit /etc/pam.d/common-session and add

session required pam_mkhomedir.so skel=/etc/skel umask=0022

Apply changes

sudo update-rc.d nslcd enable

Configure lightdm to allow user to specify a username for login

sudo /usr/lib/lightdm/lightdm-set-defaults -m true

And reboot.  If the user logs in locally or via ssh, her home directory will be created automatically.