Archive for the ‘Web Development’ Category

For those that don’t already use Last Pass! Check out this great site!

For those that don’t already use Last Pass! Check out this great site!

troll

Postfix Stats using AWStats on an ISPConfig 3, Ubuntu 14.04 Server running Nginx

So I was looking for a quick way to get SMTP statistics for the Postfix install on an ISPConfig 3 server and thought I would share the configuration for anyone else that would want to do that.

After following one of the “Perfect Server” tutorials like this one below for Ubuntu 14.04 you can configure awstats and the cron to generate some SMTP stats from your postfix mail logs.

http://www.howtoforge.com/the-perfect-server-ubuntu-14.04-nginx-bind-mysql-php-postfix-dovecot-and-ispconfig3

1. Copy maillogconvert.pl to the aws directory..

root@server.com:/root/# cp /usr/share/doc/awstats/examples/maillogconvert.pl /usr/share/awstats

2. Create awstats postfix database directory:

root@server.com:/root/# mkdir /var/lib/awstats/postfix

3. Copy the AWStats Config as a template /etc/awstats/awstats.conf to /etc/awstats/awstats.postfix.conf

root@server.com:/root/# cp /etc/awstats/awstats.conf /etc/awstats/awstats.postfix.conf

4. Edit the /etc/awstats/awstats.postfix.conf file and modify the options specified.

root@server.com:/root/# nano /etc/awstats/awstats.postfix.conf


# line 51: change
LogFile="/usr/share/awstats/maillogconvert.pl standard < /var/log/mail.log |" # line 63: change LogType=M # line 122 make it comment and add the line below # LogFormat=1 LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd" # line 153 specify hostname SiteDomain="server.com" # line 222: change DirIcons="./icon" # line 831: change like below LevelForBrowsersDetection=0 LevelForOSDetection=0 LevelForRefererAnalyze=0 LevelForRobotsDetection=0 LevelForSearchEnginesDetection=0 LevelForKeywordsDetection=0 LevelForFileTypesDetection=0 LevelForWormsDetection=0 # line 905: change your language Lang="en" # line 949: change like below ShowMonthStats=UHB ShowDaysOfMonthStats=HB ShowDaysOfWeekStats=HB ShowHoursStats=HB ShowDomainsStats=0 ShowHostsStats=HBL ShowRobotsStats=0 ShowEMailSenders=HBML ShowEMailReceivers=HBML ShowSessionsStats=0 ShowPagesStats=0 ShowFileTypesStats=0 ShowOSStats=0 ShowBrowsersStats=0 ShowOriginStats=0 ShowKeyphrasesStats=0 ShowKeywordsStats=0 ShowMiscStats=0 ShowHTTPErrorsStats=0 ShowSMTPErrorsStats=1

5. Comment out the include at the bottom of the /etc/awstats/awstats.postfix.conf file.


#-----------------------------------------------------------------------------
# INCLUDES
#-----------------------------------------------------------------------------

# You can include other config files using the directive with the name of the
# config file.
# This is particularly useful for users who have a lot of virtual servers, so
# a lot of config files and want to maintain common values in only one file.
# Note that when a variable is defined both in a config file and in an
# included file, AWStats will use the last value read for parameters that
# contains one value and AWStats will concat all values from both files for
# parameters that are lists of values.
#

# Include "/etc/awstats/awstats.conf.local"

6. Install apache2-utils with apt-get if not already installed.

root@server.com:/root/# apt-get install apache2-utils

7. Create the .htpasswd password db to protect the cgi with basic auth.

Remember the username and password you setup here.

root@server.com:/root/# htpasswd -c /etc/nginx/.htpasswd_postfix

8. Add the location to the ISPConfig NginX configuration : /etc/nginx/sites-enabled/000-ispconfig.vhost

Add this code just before the example for phpmyadmin that should be commented out...


location = /awstats.pl {
root /usr/lib/cgi-bin;
gzip off;
include /etc/nginx/fastcgi_params;
fastcgi_pass unix:/var/run/fcgiwrap.socket;
# Not needed if already in fastcgi_params.
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/.htpasswd_postfix;
}

9. Do the first run of awstats to generate the database .

root@server.com:/root/# /usr/lib/cgi-bin/awstats.pl -config=postfix -update

10. Add this line to /etc/cron.d/awstats to keep the database up to date every hour.

00 * * * * root /usr/lib/cgi-bin/awstats.pl -config=postfix -update

11. Symlink the "icon" directory to the ISPConfig web root so we can see the icons..

root@server.com:/root/# ln -s /usr/share/awstats/icon/ /usr/local/ispconfig/interface/web/

12. Restart NginX to get the new configuration

root@server.com:/root/# service nginx restart

13. Visit the ISPConfig 3 Web UI

You should now be able to visit your ISPConfig 3 web interface and after authenticating view the parsed log statistics.

An Example URL : http://server.com:8080/awstats.pl?config=postfix

You will be prompted for the username and password you set up earlier to view the statistics.

Installing Redmine 2.1.x on Ubuntu 12.04 for an ISPConfig 3 Client

This tutorial explains how to install Redmine 2.1.x for one of your clients in an ISPConfig 3 server setup running on Ubuntu 12.04 Precise Pangolin. It does not explain how to install ISPConfig 3 on Ubuntu 12.04 Precise Pangolin. An excellent tutorial on the steps involved to do that can be found here.

1. Create the Client account in ISPConfig and a new Site

Once you get ISPConfig running correctly you should log in and setup your “Client” account in ISPConfig and then create a new “Site” to run the Redmine installation for this client.

2. Install the .deb packages to support Redmine 2.1.x

Next we start by installing the required ruby / rails and apache2 modules needed to run Redmine 2.1.x. These instructions assume you have root ssh access to the server but they could easily be run by a user with the correct sudo privileges.

# apt-get install ruby-full libmysqlclient-dev libmagickwand-dev libapache2-mod-passenger

3. Download Redmine 2.1.x

Navigate to the directory for the new “Site” that was setup. You should of course change client1 and web1 for the appropriate client number in your ISPConfig 3 setup.

# cd /var/www/clients/client1/web1/

At the the time of this writing the latest Redmine 2.1.x release was at 2.1.2 so we download the tar/gz from the ruby forge website, unzip it and make it the “web” directory of your new Site.

# wget http://rubyforge.org/frs/download.php/76495/redmine-2.1.2.tar.gz
# gunzip redmine-2.1.2.tar.gz
# tar -xf redmine-2.1.2.tar
# mv web web_orig
# mv redmine-2.1.2 web

4. Create the Database for the Redmine installation

Now you should create a new mysql database for your client’s Redmine installation in ISPConfig 3. Remember the username and password you setup for this database as you will need this for the next step when you change the Redmine database configuration.

5. Copy the database configuration example file to a new configuration file

# cp ./web/config/database.yml.example config/database.yml

6. Edit the new database configuration file and change the following values for production: to the database you have setup in ISPConfig 3.

production:
adapter: mysql
database: c1_redmine
host: localhost
username: c1_redmine
password: <enter-the-password>

6. Now we need to ensure that we have installed rake and the ruby bundler package.

# apt-get install rake ruby-bundler

7. Now go to the config directory and run the following..

# cd /var/www/clients/client1/web1/web/config/
# bundle install --without development test postgresql sqlite

This should out put something similar to the following…

Fetching source index for http://rubygems.org/
Installing rake (0.9.2.2)
Installing i18n (0.6.1)
Installing multi_json (1.3.6)
Installing activesupport (3.2.8)
Installing builder (3.0.0)
Installing activemodel (3.2.8)
Installing erubis (2.7.0)
Installing journey (1.0.4)
Installing rack (1.4.1)
Installing rack-cache (1.2)
Installing rack-test (0.6.2)
Installing hike (1.2.1)
Installing tilt (1.3.3)
Installing sprockets (2.1.3)
Installing actionpack (3.2.8)
Installing mime-types (1.19)
Installing polyglot (0.3.3)
Installing treetop (1.4.10)
Installing mail (2.4.4)
Installing actionmailer (3.2.8)
Installing arel (3.0.2)
Installing tzinfo (0.3.33)
Installing activerecord (3.2.8)
Installing activeresource (3.2.8)
Using bundler (1.0.15)
Installing coderay (1.0.7)
Installing fastercsv (1.5.5)
Installing rack-ssl (1.3.2)
Installing json (1.7.5) with native extensions
Installing rdoc (3.12)
Installing thor (0.16.0)
Installing railties (3.2.8)
Installing jquery-rails (2.0.3)
Installing mysql (2.8.1) with native extensions
Installing net-ldap (0.3.1)
Installing ruby-openid (2.1.8)
Installing rack-openid (1.3.1)
Installing rails (3.2.8)
Installing rmagick (2.13.1) with native extensions
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.

Make sure the above line is show before continuing…

8. Create the secret token.

# rake generate_secret_token

9. Initialise the Redmine Installation

# RAILS_ENV=production rake db:migrate
# RAILS_ENV=production rake redmine:load_default_data

You should see the output from creating the database and also a message stating that the default data has been loaded.

10. Make sure that passenger is enabled for apache2

# a2enmod passenger

11. Create the appropriate permissions for some directories in the Redmine setup.

# chown web1:client /var/www/clients/client1/web1/web/ -R
# chmod 755 /var/www/clients/client1/web1/web/
# mkdir /var/www/clients/client1/web1/web/public/plugin_assets
# chmod -R 777 /var/www/clients/client1/web1/web/public/plugin_assets
# chmod -R 777 redmine/files

12. Override the default DocumentRooT for your Site

You do this by adding the following to the Apache Directives section of your ISPConfig Site configuration.

DocumentRoot /var/www/clients/client1/web1/web/public/

When ISPConfig’s cron update runs and apache2 restarts you should be able to go to the URL for the Site you have setup and you should see your Redmine installation.

Hope this helps those wanting to run Redmine for their ISPConfig clients 😀 Of course keeping this up to date is another story 😉


This tutorial uses the following articles as as a references…

Moving an app from CakePHP 1.3 to CakePHP 2.0 … quite painless actually

Over the past year and a bit I have been working with Monash University to develop a web based learning tool for their School of Pharmacy. The project is called “MyDispense” and after some initial twists and turns working out which tools we were going to use to deliver the job, our team eventually settled on the CakePHP Framework (version 1.3) 😀

Eventually the project hit a third development cycle and this coincided with the CakePHP team’s stable release of version 2.0. We were planning some much needed refactoring and D.R.Ying up anyway and naturally wondered if we should move the app to the new version of the framework while we were at it? There were of course some nice new things in CakePHP 2.0 that looked good, not to mention it was considered faster due to the lazy loading of models and removal of php4 support. So we thought what the hell and made the choice to move to CakePHP 2.0. A couple of small gotcha road bumps did crop up due to some differences in Cake 2.0 and 1.3 but they were quite easy to solve as you will see.

Okay so we made the decision but where do we start? Of course we started with some google searches 😛 We looked for tutorials on the web. Some advice from brave peepz that have gone before us and might save us some time and headaches. There wasn’t much out there saying people had trouble so that was a good sign.

Our first port of call was the CakePHP 2.0 Migration Guide, for obvious reasons 😛

I read this page and of course got to the “file names” section and my first thought was … “f#@k we were going to have to modify quite a few folders and file names for this migration”. My second thought.. “erm.. wait a tick… doh! Why don’t we just use the ‘bake’ tool from the Cake Console and bake it all out?” So my colleagues and I stripped the database of unwanted tables and redundant stuff and set to the task of baking out the models, views, admin views and the controllers using the console in Cake 2.0.

One little “gotcha” for me using the bake command in 2.0 was “which directory I should execute the “cake bake” command from?”. After a little bit of head scratching I realised you should do this from the “app” directory as your working directory. I would ssh to my “app” directory on my development setup and type “Console/cake bake”. This produced the familiar interface for selecting what I wanted to “bake” without throwing any errors and of course would put the new files in the correct Model, View and Controller directories in my application.

The next “gotcha” was in regards to the way MyDispense’s dynamic PNG images were displayed in the browser. Some changes to the way that the CakeResponse api works required us to add the “$this->response->send();” call after after setting the PNG image header. In version 1.3 of Cake this wasn’t a requirement.

<?php
 $response->header(array('Content-type: image/png'));
 $this->response->send();
?>

Another issue that came from the move (which we should have checked earlier really) was that some of the older components we relied on in the version 2 of MyDispense was build for the old CakePHP architecture and had not been brought up to date. The first of these was the CAS Auth component we found here. If we had time I would have modified the existing component into one that 2.0 can use but unfortunately this was not possible. Instead I came up with what I think is a more flexible solution using phpCAS directly. I will blog about this process soon.

So in the end quite a painless process and moving a cake app for 1.3 to 2.0 was worth. If you have the time I recommend doing it. Our application is measureably faster and happier for it 😀

Sending email from command line using PHP

Here is a simple php script that sends an email from the command line using php. It sort of assumes you have the PHP CLI installed (“#apt-get install php5-cli” on debian). The aim is for a quick way you can test if your localhost SMTP server is sending emails from PHP. Of course there are some more sophisticated classes written if you need to do something fancy.

#!/usr/bin/php
<?php

/* USAGE: #./.php test@email.com 'this is a bit of text' */

$headers = 'MIME-Version: 1.0' . "\n";
$headers .= 'Content-type: text/html; charset=iso-8859-1' . "\n";
$headers .= 'From: Command Line' . "\n\n";

echo "Sending mail to ".$argv&#91;1&#93;."...\n";

if (mail($argv&#91;1&#93;, 'test email', $argv&#91;2&#93;, $headers)){
  echo 'success!'."\n";
}else{
  echo 'failed!'."\n";
}

exit;

?>

Cannot delete or update a parent row: a foreign key constraint fails

Problem deleting data with FOREIGN KEY constraints in MySQL….

Ah yes. Your working on that system that uses the MySQL FOREIGN KEY constraint on some tables.

You want to truncate a table but get an error message because you have already truncated the data in the related table.

“Cannot delete or update a parent row: a foreign key constraint fails”

Fear not you can disable the constraints in the mysql shell…

mysql> SET foreign_key_checks = 0;
mysql> TRUNCATE some_table;
mysql> SET foreign_key_checks = 1;

Think Before You SymLink

I was recently working on moving some PHP software from one server to another for a client of mine and ran into a spot of trouble.

The previous developer that was maintaining the software and the “production” environment is a mate of mine so I won’t bag him too much in public, although he knows who he is and the pain he caused me .. lol ^_^

Anyway suffice to say the move did not go 100% according to plan. Even with many years of experience deploying websites and running Linux servers, I was not really ready for what I was about to face.

You see I originally installed this Debian server many moons ago (back in the day when Q-mail Rocks was the set up to have) and in good Virgo fashion I left it in a very nice and ordered state in the hands of my mate. The server subsequently had a couple more guru Linux cooks brew a few concoctions up on it.

Anyway the point of all this drivel is that my mate and the other cooks in their infinite wisdom had decided that the humble symbolic link was the weapon of choice to ensure proper and extensive (and I mean extensive) “code reuse”.

Let me tell you it was like moving a house of cards from one table to another.  In there were symlinks that linked to other symlinks which in turn were symlinks to development files. WTF???

There were image directories that had symlinks to other image directories which in turn had symlinks back to the original image directory. You should have seen the size of the tar files it was trying to create! Anyway lets just say an accurate description would be a bowl of spaghetti crossed with a spider web made by mr crack spider.

I am still finding missing symlinks but have since found this cool bit of code that re-curses through the directory tree and finds all the broken symlinks.. yay!!

for i in `find ./ -type l`; do [ -e $i ] || echo $i is broken; done

Anyway there are plenty of better ways to re-use code kids. Symbolic linking is not really the most maintainable option!

Think before you symlink!

pushLMS

pushLMS as bold as the name suggests  is an attempt at developing a type of Learning Management System.

pushLMS is the name of a new project I am working on with the crew at OSVR, Monash University Pharmacy and Pharmaceutical Sciences and Victoria University.

Combining Virtual Worlds like Open Simulator and Second Life with some PHP software written using the cakePHP MVC framwork, we are creating what could be called “Virtual Education Environments” in the form of assessment for the Students.

We aim to eventually make it open source (well that is what OSVR is about) and I will no doubt blog a bit more about this as the software develops.

More info on this project can be found at the pushLMS google blog for now.

Return top