Web Server

Apache

Install Apache 2, use SFTP as it uses SSH, so no need to install pureFTP, or similar.   https://help.ubuntu.com/community/PureFTP

https://www.raspberrypi.org/documentation/remote-access/web-server/apache.md Continue on to install all the requirements for WordPress.  https://www.raspberrypi.org/learning/lamp-web-server-with-wordpress/worksheet/

sudo apt-get install apache2
sudo chown pi /var/www/html
# enable rewrite mod
sudo a2enmod rewrite

PHP

sudo apt-get install php5 libapache2-mod-php5 -y

MySQL

sudo apt-get install mysql-server
# enter a password for mysql root user
# install ancillary packages 
sudo apt-get install mysql-client php5-mysql
# restart apache2
/etc/init.d/apache2 restart

phpMyAdmin


apt-get install phpmyadmin
# select which http server to configure (apache2)
# configure database with dbconfig-common (select yes)
# provide mysql admin user password to install phpmyadmin
# provide mysql password for phpmyadmin 
# configure apache to work with phpmyadmin
vi /etc/apache2/apache2.conf 
# add the line Include /etc/phpmyadmin/apache.conf
# restart apache
/etc/init.d/apache2 restart

Log Files

Apache2 provides an access and an error log in the /var/log/apache2/ directory.  access.log  contains all details of requests processed by the server including the IP address the request originated from, timestamp and user agent.   All messages and errors from the server are stored in error.log.  For example

AH00558: apache2: Could not reliably determine the server's 
fully qualified domain name, using 127.0.1.1. Set the 
'ServerName' directive globally to suppress this message

Virtual Hosts

As soon as you want to host multiple sites from the server, then it is necessary to work with the configuration files.  Apache2 loads configuration data from individual files as an alternative to editing a long configuration file which may introduce errors.

# create a file in /etc/apache2/conf-available/
sudo vi /etc/apache2/conf-available/servername.conf
# this will remove the AH00558 error above

# copy default site file
cd /etc/apache2/sites-available/
cp 000-default.conf test.domain.conf

# edit test.domain.conf, specify ServerName and DocumentRoot
vi test.domain.conf
ServerName test.domain
DocumentRoot /var/www/test/

# create DocumentRoot and adjust permissions
mkdir /var/www/test
chmod -R 755 /var/www/test/
chown -R www-data:www-data /var/www/test 

Domain Name Troubleshooting – https://lookup.mxtoolbox.com/domain/blueshrapnel.co.uk/

SSL

Benchmarking

fail2ban

Other links

Remote Access with VNC on Raspbian

Virtual Network Computing (VNC) is a widely used remote technology, particularly in heterogenous networks.  For this reason, several open source VNC projects exist, including RealVNC and TightVNC.  VNC enables one to control a number of different computers from one keyboard.  It is different to a remote terminal session such as SSH as you do not log in to a server.  Anything you do on the VNC session is as if done by the user currently logged into the remote desktop.

VNC requires a client and server to create a session.  The server runs on the remote desktop and open a vncviewer on the client.  VNC estalishes remote access over either a local arean network or over the internet using TCP/IP, implementing a Remote Frame Buffer (RFB) which grabs the screen image and sends it to the client.  The client displays the remote screen in a window on the client desktop.  The client transmits mouse and keyboard data back to the server.

VNC creates stateless sessions, enabling the user to disconnect and reconnect from different machines irrespective of which operating system is installed on the client or server.

Security

However RFB does not travel over a normal connection in an encrypted mode.  Consider using OpenSSH and use VNC through an encrypted tunnel.

Raspberry Pi Specifics

The first time you run a VNC server it will prompt for a system password. The server will appear as hostname:1, hostname:2 etc.  The server chooses the first available display number and tells you what it is.  Setting the DISPLAY environment variable can cause applications to use a specified display.

RealVNC

As of September 2016, RealVNC software is available for free for academic and non-commercial use on Raspberry Pi’s.  Note that only one VNC server can be installed at one time.  So if you have installed tightVNC previously, you will need to remove it.

# remove tightvncserver
sudo dpkg -r tightnvncserver

# install realVNC 
curl -L -o VNC.tar.gz https://www.realvnc.com/download/binary/latest/debian/arm/
tar xvf VNC.tar.gz
# check the name of the file, it will look like
sudo dpkg -i VNC-Server-5.3.2-Linux-ARM.deb
# Raspberry Pi licensing automatically 

# configure raspi to start realVNC when powered up
sudo raspi-config
# select Advanced Options/VNC and set to yes

TightVNC

# install tightvncserver
sudo apt-get install tightvncserver
tightvncserver
New 'X' desktop is hostname:2
starting applications specified in /home/pi/.vnc/xstartup
Log file is /home/pi/.vnc/hostname:2.log
tightvncserver -kill :2

References:

Raspberry Pi Cluster

A photo of the Southampton University Raspberry Pi Cluster.The first time I heard about a computer comprised of multiple Raspberry Pi’s, it was the Southampton University ‘Lego’ Computer.

I discovered a blog entry which described a Raspberry Pi cluster built by Nnvidia High Performance Computing Engineer, Adam DeConinck Some technical details about Adam DeConinck’s pi cluster.  Due to public demand, they made some details available, including a readme file on github with more information.

 

Load Balancing

Load balancing is very possible for the Pi.

There are many different ways to do this, one is to split services (webserver,mysql, ftp) over different Pi’s. Another is to split the files across Pi’s (images on one, html on another etc). But in my opinion, the best (and most complicated) way to achieve load balancing is to setup one Pi as a nginx proxy to distribute the load to the two Pi’s hosting the same files. You can even set how much load is distributed to each Pi!

Hadoop

“The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.”

– http://hadoop.apache.org/

Basic Modules

  • Hadoop Common – libraries and utilities needed by other modules
  • Hadoop Distributed File System (HDFS) – stores data providing very high aggregate bandwidth across the entire cluster
  • Hadoop Yarn – managing resources and scheduling users and applications
  • Map Reduce – programming model that scales data across the processes

Hadoop Distributed File System

Distributed scalable and portable file system written in Java for the Hadoop Framework.  Includes secondary NameNode which connects to primary NameNode and builds snap shots of directory structure.

MapReduce Engine

Different ways to submit and track jobs. Overhaulled in Hadoop 2.3 MRV2 referred to as YARN.  Split up the major functionality of job tracker, resource manager and scheduler. YARN is compatible with MapReduce.

Started as Google Bigtable and MapReduce / Google File System (GFS).  Wanted to be able to access the data with a SQL style language.  Facebook, LinkedIn and Yahoo stack all had some similar products in their stacks.

 

 

What to do with my Raspberry Pi’s…

A photograph of a fresh installation of Raspberry Pi with some Linux books.
Rounded up all the Raspberry Pi’s and Linux books from around the house.

Most of the time with Raspberry Pi’s, I look at them and think it would be nice to do something really cool…. but then real life takes over and the moment is gone.  I’m not alone in this, but now with refreshed determination…. and the fact that I have an Operating Systems and Network project to complete on anything to do with Linux… my plan is to build a cluster of Raspberry Pi’s and experiment with Hadoop and see where that takes me, initially operating the environment within a dedicated local wireless network.

First thing to do is install emacs… get some system info and practise shell scripts by way of revision for the course, just waiting for…

sudo apt-get upgrade
sudo apt-get install emacs (or your preferred editor)
// expand your SD card storage
sudo raspi-config -> menu option Expand FileSystem