I was aiming to use the hardware driven serial on the Arduino’s ATMega328 chip – the reference docs claim it can support custom baud rates, so I figured it should just work.

The code base over at OBDuino suggests that Serial.begin(10400) should also work. I was blaming the circuit used for the lack of communication, which would make sense, but it does work with NewSoftSerial. I tested this last night by using 9600 baud with an ELM327 cable. The test fails using the baud rate of 10400 in the Hardware serial.

With NewSoftSerial working at 9600 baud, the missing link then becomes how to use 10400 baud with NewSoftSerial.

The answer was simple enough thankfully:

#if F_CPU == 16000000

static const DELAY_TABLE PROGMEM table[] =
{
// baud rxcenter rxintra rxstop tx
{ 115200, 1, 17, 17, 12, },
{ 57600, 10, 37, 37, 33, },
{ 38400, 25, 57, 57, 54, },
{ 31250, 31, 70, 70, 68, },
{ 28800, 34, 77, 77, 74, },
{ 19200, 54, 117, 117, 114, },
{ 14400, 74, 156, 156, 153, },
{ 10400, 106, 218, 218, 215, },
{ 9600, 114, 236, 236, 233, },
{ 4800, 233, 474, 474, 471, },
{ 2400, 471, 950, 950, 947, },
{ 1200, 947, 1902, 1902, 1899, },
{ 300, 3804, 7617, 7617, 7614, },
};

The baud rate 10400 is simply 10400 / 9600 = 1.0833333. Then for each of the 9600 rx and tx values, divide by 1.0833333.

Testing this with the OBD wake up that gets sent every second or so (ox01, 0x3E, 0x3F) gets the correct data as it would be at 9600 baud (and matches what I tested with the ECU in the car).

It’s not ideal though, software serial is using CPU clock cycles to fake an actual serial port, there needs to be a reason as to why the Arduino didn’t like 10400 baud, this is against the knowledge that it works with the OBDuino – a project that’s been used numerous times in the USA, and would therefore be successful.

It really should come down to whether the circuit is correct (I’ve tried the variation here; http://blog.perquin.com/prj/obdii/, and here elmelectronics.com/DSheets/ELM323DS.pdf, and the very simple zener and resistor circuit here, http://sterntech.com/obdii_protocols_iso.php) – not to mention variations to those to see if it was a TX related issue (- I was leaving TX empty for receiving only).

Being the holiday period it’ll take too long to track down one of the IC chips used in the OBDuino, and the MC33290 IC used there is End of life.

I’ve looked over this circuit by someone else who tried the same trick as the first link above, but that’s really only adding protection, and looking through his blog, I doubt he found a result – he claims he got the initial init – but that’s it)..

Logically thinking, the serial would be processed as one might expect, the data bits are logic high and low – 5V high, 0V low, the standard used in the car is that the circuit idles high and goes low when bits are being sent – so 0, 1, 0, 1 is simply 0V, 5V, 0V, 5V. That should be received, but perhaps the arduino’s software is discarding it – though that makes less sense compared to the circuit not producing enough current to trigger a logic high / logic low as need be. I don’t think the pulses are long enough for the DMM to pick up either.

I’ll have to toss up on how to solve this issue another time, I’ve got something to work with now that’ll get me the test setup I need for data transmission.

I tried a few methods of discovering just what the ones and zeros were telling me, and tonight, a break through, I now know precisely the 5 bytes used to initialise the Nissan ECU for Consult II protocol (and the ECU reply)!

The next is to then get discovering the raw data. The bytes always require calculation to derive the real value (i.e. Hex: 0A is what might be returned for RPM – but you won’t find a car doing 10 RPM). Of course fishing out what the PIDs are and those calculations is going to be fun.

I do hope to discover most of the information available and gather it then set up a microcontroller and small LCD display with it for data logging purposes.

That can then store info on real time fuel economy, giving the driver feedback as to whether the current style matches desired consumption (important with fuel prices catching reality). It has other reasonable uses too, log speed and attach a GPS and you can get some idea of what the car was doing and where. Get cameras and the above GPS and you can log useful information there too.

That’s off in the distance, I’ve still got to get the connection going, get the bytes moving and get the code written.

But on that note, why is it that when the simple solution is staring me right in the face (literally in some cases), I take the longest road?

It becomes a different thing to work with when you are trying to calculate dynamically with it. It’s not overly difficult, it just has the potential to make me stop in my tracks and think about the logic being used.

It happened today. I started this week hoping to get to work, finish off one project inside of a short time, and move on to something different. Monday was severely flooded, I couldn’t get access to work – to uhh work. Roads were flooded. Half the day wasted on account of the rain.

I started today hoping to finish what I was working on, and by mid after noon began pondering how I’d make the ability to use two date ranges work together.

There are two date ranges, one represents the set of collected data, the data range. The other represents the variable(s) applicable to that data set – the variables are date ranges.

So to make them work, I come across issues such as – how do we work with the date of the variables being ‘after’the reporting data set (a very real possibility, I anticipate seeing the variable date ranges begin and end with overlapping data periods.

Thinking more of it tonight, I figured this is the best representation of what must be done (in logic terms):
if (dataDateStart < variableDateStart) then
if (variableDateEnd < dataDateEnd) then
//days involved is the days between variableDateStart and variableDateEnd
else if (variableDateEnd >= dataDateEnd) then
//days involved is the days between varaibleDateStart and dataDateEnd

else //if dataDateStart >= variableDateStart)
if (variableDateEnd < dataDateEnd) then
//days involved is the days between dataDateStart and variableDateEnd
else if (variableDateEnd >= dataDateEnd) then
//days involved is the days between dataDateStart and dataDateEnd

That gives the complete desired overlap needed when considering the variables.
If the variables don’t meet the data date ranges, that becomes a user error (otherwise the loop fetches multiple records with multiple date ranges and acts accordingly).

It’s not overly difficult in the basic form, I just don’t always see the simple when thinking of how to make it work.

The first is the Solar Hot Water system, which is where I wanted to start – wirelessly read some temperature sensors, and send them to the server for logging purposes, and set up a warning to turn on electric heating if need be (it’d be a relay if it could, but it can’t at the moment).

The original point was so we could gauge some idea of how much hot water was flowing out to the flat compared to the house, but then looking at the system since we’ve again emptied the flat, the solar hot water is easily serving our requirements – it’s obvious the source of waste was the flat. Still would be useful to have wireless information on.

Next, I have some thoughts on monitoring the moisture and temperature of the soil, and then have a pump trigger as appropriate, to find and keep them in the ideal growing range. Not enough time to put that together in a good way.

Wireless data is easily done with the Arduino microcontrollers and the NRF24L01 from Nordic, and another use which I have just started putting together is wireless data transmission – I’ll be hooking it up to the serial line in the car to get that data wirelessly transmitted whilst I get the protocol data. Store this, read it, and then it’s just a matter of sending that data out correctly, to get the same data in reply.

The difficulties are not in the circuit (though this can be somewhat not so simple to put together), it’s not in the code (that’s easy, even if you do forget to allow for some events), it’s in the enclosure – putting them together in a way that allows the board, battery, and additions in a neat manner is not so easy.

Creating a custom board with the microcontroller and wireless chip on it would be a better idea – with inputs and power planned in the design.

After much, much, much Googling and some testing, I concluded that there is no OBDII support, and the car has only Nissan’s Consult II protocol available.

That’s bad in terms of support for performance monitoring, as the protocol for Consult II doesn’t seem well documented (and those who do have it, don’t seem to be detailing it – rather going to lengths to conceal their own findings).

There’s always more than one way to skin a cat, and I was reminded of that again today, in trying to solve another issue. The original intention was to get real time data on fuel – and I can do that just as I did in the Sonata – injectors get turned on and off, so use that signal. The car has a working speedo, use that signal.

I spent hours upon hours over weeks searching and found as much as I could about the protocol available – I flicked through the service manual and of course it’s there in black and white!

Getting data from Consult II is not out of the question though – bit banging is a well documented technique for discovering protocols and the data being carried across them. It’s a fair bit of effort though.

I tried to test for OBDII specifically, but that didn’t work. The battery in the car went flat again too, and I was really only in it for 30 minutes or so with the music and sometimes the light on. 30 minutes with such a low load (stereo couldn’t be too much!), suggests either the speakers in the back are very high, or this battery is needing some maintenance or investigation. Luckily I have jump start cables from the last flat battery, and the Sonata is still lingering around.
Now it’s out on the street (in case I need another jump start, it’s difficult moving a car around a tight corner), maybe after we finish getting the Tyres and Servicing on the Pulsar, we’ll find a new home for it.

Ignoring that, we’ve been planning some extra fruit trees for some time now, we were waiting on some trees ‘taller than the seller’to become available at the local markets over several weeks – we gave up waiting and got some from the nurseries instead which seem taller than us.

Our choices were limited primarily to what we know the kids love to eat. The vegie patch has been pretty bare so we moved the garden beds from the back corner of the yard and bought them forward more, making room for the trees to go in on the side and planned spacing between them allowing for growth.

This created it’s own problem – the Citrus trees need to go in the ground from their temporary homes (a great idea at the time). They can’t now as they are all putting on rapid growth and any transplanting would likely cause them harm.

The space available for the trees is limited, so the shed will need to go for that to happen – which is already planned.

I didn’t see it was on as it was raining last night and I was too focused on getting the oldest kid inside.

I tried to clutch start it, figuring it was just the interior light, couldn’t be that flat, surely.. stuck it in 2nd, rolled down the driveway, but just nothing there.

I’ve been planning on either selling off the Sonata recently (fully disclosing every single issue with it), to deregistering it and leaving it there to rust out or getting it to scrap metal.

Thankfully progress has been slow, and so the car is still registered. It’s battery which was sure to be flat was surprisingly capable of starting the car – and no more belt squeal for some reason. That’s probably related to the trip out to the dump last weekend – though it needed a charge to make that trip.

I was tossing up between charging the battery in the Pulsar overnight or getting jumper cables and taking it for a spin to get the battery charged – speedy jumper cables won, and worked surprisingly well (I was expecting to have to get the Sonata revved up – but not so).

Fuel was low (as expected, I wanted to run the fuel down in the Sonata so it didn’t sit there with a lot of fuel), I drove it 20ks to work and figured it probably would run out of fuel on the way home at some point – so had to add more fuel. I’ll burn it off somehow.

The Sonata stinks too though, it had a bad leak on the passenger side, and pretty much had a lake on the passenger side that soaked the carpet – it stunk and still does.

Back to it though, I’ve been trying to find a feasible method of communication with the ECU of the Pulsar. The Sonata had used MUT (Mitsubishi protocol), for data. Nissan have their own Consult II protocol, which doesn’t seem to have much on the net in terms of how it operates.

OBDII is non existent I think. So I’m looking at other communication methods so I don’t have to solder on to the injector and speedo wires as I did in the Sonata for the Carduino. The Data connector exists, perhaps OBD II is there but the cable I used to connect was not functional – so I am going to attempt that.

Then I will sus out if there is “CAN” communication – “Controller Area Network” communication, this allows the ECU to talk to other parts of the car – such as Transmission in Auto cars, or in the case of ours, the Combination Meter (Odometer display) also would be talking to it. If I can chase that out, I can run that to the OBD II connector and that would give me communication to the ECU.

I dislike having to use the remote to skip tracks / set random mode that our head unit has – so I want to rig up something with that to send signals directly, mounting some buttons above the unit.

Then I want to add an annoying speaker to the interior light, thus if it’s on, it’ll be damned annoying, and only turn off with the doors shut (or turned off), thus preventing it happening again.

The LCD supports loading of custom characters in addition to the standard character set – with Arduino this seems very easily done.

However the Launchpad, based on the msp430 chip from TI is still relatively new – so not a lot exists at the moment for setting up with Launchpad and getting custom characters loaded.

Thankfully there is already a large example out that has the basics of the LCD done. What was wrong from that basic setup was the ability to work with the newer msp430-gcc (uniarch) compiler – from here. I moved to this some time ago to use the full benefits of the msp430g2553 chip – it is the more beefier chip that is a drop in replacement for the included chip (also features onboard UART, great for serial comms).

The basics needed changing, “Wiring.h” had to be edited to be compatible with the new compiler, the pin out was different (the schematic at the URL above was missing contrast).

The changes made to Wiring.h are:
//#pragma vector = TIMERA0_VECTOR       
interrupt(TIMER0_A0_VECTOR)  TA0_ISR (void)        {       
//      __low_power_mode_off_on_exit(); // Restore Active Mode on return       
}

The low power mode is commented out – I won’t need to use it for the chip is low power to begin with and will be connected to mains via USB – but it’s very easy to modify the SR register to take out sleep mode.

#include <stdbool.h>
#include <stdio.h>
#include <msp430.h>
#include <legacymsp430.h>
#include <stdint.h>
#include <msp430g2553.h>// Headers for specific device#
include "HD44780LIB.h"

To design a character, there’s a handy webpage here that explains the bits that need to be written to the display (8 possible custom characters). I wanted numbers and after having the MPGuino code handy from my ongoing project for the car – I simply converted those bits to hex and used them:
unsigned char chars[] = { 0x1F,0x1F,0x0,0x0,0x0,0x0,0x0,0x0,
0x0,0x0,0x0,0x0,0x0,0x0,0x1F,0x1F,
0x1F,0x1F,0x0,0x0,0x0,0x0,0x1F,0x1F,
0x1F,0x1F,0x1F,0x1F,0x1F,0x1F,0xF,0x7,
0x0,0x0,0x0,0x0,0x0,0xE,0xE,0xE,
0x1F,0x1F,0x1F,0x1F,0x1F,0x1F,0x1E,0x1C,
0x7,0xF,0x1F,0x1F,0x1F,0x1F,0x1F,0x1F,
0x1C,0x1E,0x1F,0x1F,0x1F,0x1F,0x1F,0x1F };

The characters are rounded characters, they look good on the display. To load the characters, use the command and write functions as below:

HD44780_init(&theHD44780,4,3,0,5,2,1);        //this is different, I copied the Wiring functions to use P2 (Port 2).
HD44780_begin(&theHD44780,16, 2);       
HD44780_print_string(&theHD44780, "Hello");       
HD44780_pulseEnable(&theHD44780);       
HD44780_command(&theHD44780, 0x48); 
for (i = 0; i<64; i++) {
                HD44780_write(&theHD44780,chars[i]);
  }
HD44780_pulseEnable(&theHD44780);              
HD44780_command(&theHD44780, 0x01);       
HD44780_pulseEnable(&theHD44780);       
HD44780_command(&theHD44780, 0x80);       
HD44780_gotoXY(&theHD44780, 0, 0);

The init line differs because I changed the pin outs so I could use the UART on the chip (P1.1 and P1.2). I also had issues with the display not loading properly so I moved all to Port 2 and all seems well now. In the above, I also used 0×48 as the CGRAM command – it’s noted in the original as being 0×40, the 0×48 request worked – I sticked with what worked.

The gotoXY function I added t HD44780LIB.h, it is below, but is sourced from MPGuino – it’s a simple function:

void HD44780_gotoXY(HD44780 *me, unsigned int x, unsigned int y) {
                unsigned int dr=x+0x80;
                if (y==1) dr += 0x40;
                if (y==2) dr += 0x14;
                if (y==3) dr += 0x54;
                HD44780_command(me, dr);       
}

Accessing the characters is easy enough – HD44780_print_byte(&theHD44780, 2) would display character 2 from the CGRAM (which, if we start from 0 is the third character in CGRAM).

The end result being targetted is adding this LCD to the system I have in the lounge room, it has a backlight, so I’ll display the clock and a few other things that aren’t easily seen with just the TV on. I’ll cut an opening for it in a cover in the 5.25″ bay, and glue some stand off screws to hold the display upright and bury the MCU down the bottom, and tap USB from the Motherboard directly.

I’ll be adding some other functions to it as well – Now Playing comes to mind for streaming music.

There is already an LCD project that exists for Linux completely negating the need for the MCU in the middle – it’s here – but I already had the LCD rigged up to the Launchpad when I decided to start it – so I’ll stick with it.

Data Validation is a very, very, very important aspect of any user facing application, be it a website, be it a program, be it a simple calculator. The data must be validated to ensure it conforms to the expected input, prior to being used for logic processes and resultant output.

So, when I recently went through some data and saw such elementary and simple things like postcodes not being checked against ‘states’and suburbs spelt incorrectly, I was immediately thinking – who the heck gets paid to write such garbage..

Unfortunately, that is the case, and as is the reality in many workplaces, one which isn’t easily overcome.

I’ve always held the view that a user, be it malicious, incompetent or just plain lazy, can and often will supply applications (website pages, actual programs, scripts, anything) with garbage data.

The responsibility for testing the validity of that data lies wholly with the application author, and where they fail to do that, they’ve failed in the development of that application.

.. Another reminder of the complete incompetence of some of the people who somehow manage to hold jobs in Australian workplaces for so long.

I’ve got many choices when it comes to putting something together for a particular purpose, for example, recently, I’ve been using bash to handle simple tasks.

PHP was my previous language of choice for anything, complex or simple – it’s a great scripting language of course, and more capable then a bash shell script. It’s mostly used as a web server scripting language, but that never stopped me making several monitoring scripts, and some asterisk AGI scripts using it. I’ve also used it for other purposes like any shell – working a list of 200 domain names at one point for a specific task – wasn’t worth writing a file.

Then there is Perl, another good language. I’ve dealt with it before, and could still create with it. I haven’t used it recently, because of a lack of purpose, I don’t have a need for it – but much of what I’ve done in PHP could be replicated in Perl, and likely better. I do have a perl based IRC trivia bot that is reasonably solid – I’ve hacked on some MySQL concepts to it as well.

I’ve also worked with Visual Basic and VBScript – Microsoft specific languages, incompatible with Linux without resorting to Wine is a serious negative – but on the other hand, it’s a very readable language.

Java and Javascript – I’ve not dealt with Java before, and have no plans to, however Javascript, has always been a must to know, and with the introduction of AJAX and other similar web browser interactive concepts, it remains a very relevant language.

We have .Net (the reinvention of visual basic, introduction of C#). I have no idea why anyone would desire .NET, it’s not cross platform, it’s another language to learn that could have an expiry date any day now (just look at the original Visual Basic.. ), and closed source. I’m certain people have developped some great applications with it – I’ve used it once, for a payments page, the IDE was nice, the code read well – but I’d have rathered PHP – which is cross platform  and doesn’t seem to have a use-by date.

Then, I’ve now recently come across Python, and it’s a fantastic language, it’s cross platform. In just this week I’ve managed to make a simple application listen for UDP packets on broadcast, and send out libnotify alerts on my ubuntu machine. I use it so I don’t need to run netcat and monitor it there!

I’ve dipped into C/C++ before – it’s a far more proven language, being around for a much longer time and still in wide use (oh, and it’s cross platform). It’s not very readable, but I’ve heard around circles that if you can learn C / C++ you can learn Java. The structure of the code is very similar.

There are many different languages out there, from my view, I’d have thought you’d need to be multilingual.

I now ponder, for those who seek to start a true career in programming, what language do they master – why would they?

For me, I’ve got no problems picking up languages, but to master one is to commit to it for a longer term – and to me, Python is there, Perl is there, PHP is there, Bash will be there. .Net certainly will not be there – it’s too much of a risk.

With open source languages, and the fact they work cross platform (meaning I can load up PHP on a windows server, and a Linux server), there’s no reason to doubt the viability of  the language.

There’s not many ways one can set about mastering any particular language however, most industries I can think of will be using a combination of many – and we can’t all master all those languages – only to have them replaced.

I could ignore the statements that fail – but that could be potentially dangerous – leaving with no slave backup at all.

The next best option to ensure the database is up to date is simple, monitor mysql for slave errors, which is what I do now – on my home server, I have it querying the 4 services, and checking if the two “Yes” options are ever “No”.

I created the script to run in bash, I usually heavily use PHP for all sorts of tasks (yep, many years ago, I did a PHP script for simple ping monitoring, and just stuck with it).

Using bash doesn’t require as many resources (not that PHP is resource intensive).

The script is probably pretty simple, but it was a bit of a learning curve for me (with my minimal bash experience).

/usr/sbin/mysql_monitor:

#!/bin/bash
EMAIL=”admin@email.address”
HOSTS=”192.168.x.x 192.168.x.x x.x.x.x x.x.x.x”
while :
do
for host in $HOSTS;
do
result=`mysql -h $host -u root –password=x –execute=”SHOW SLAVE STATUS\G” 2>&1`
slave_result=`printf “%s\n” “$result” | grep -i running`
connect_result=`printf “%s\n” “$result” | grep -i Error`
if [[ $slave_result == *No* ]]
then
echo “$host – $result” | mail -s “SLAVE OUT OF SYNC”  “$EMAIL”
fi

if [[ $connect_result == *connect* ]]
then
echo “$host – $result” | mail -s “MySQL Connection Issue” “$EMAIL”
fi
done
sleep 1800
done

HOSTS is a variable which contains the IPs of the hosts to be monitored.  EMAIL is self explanatory.

The script sleeps for 30 minutes so that it’s not hammering, yet will check more often then my random every so often, sometimes weeks between follow up.

It’s added bonus is it’ll also tell me if a MySQL instance isn’t running by chance to ensure I am alerted to that.

I created an init script too to make sure it runs on startup on my home CentOS box.

/etc/init.d/mysql_monitor:

#!/bin/sh
# chkconfig: 2345 95 20
# description: MySQL Monitor
# Monitors MySQL Slave Status
# processname: mysql_monitor
APP=/usr/sbin/mysql_monitor
PIDFILE=/tmp/mysql_monitor.pid
case “$1″ in
start)
echo -n “Starting $APP: ”
$APP & pid=$!
echo $pid > $PIDFILE
echo “Done.”
;;
stop)
echo -n “Stopping $APP: ”
PID=`cat $PIDFILE`
kill -9 $PID
rm -f $PIDFILE
echo “Done.”
;;
restart)
echo -n “Restarting $APP: ”
PID=`cat $PIDFILE`
kill -9 $PID
rm -f $PIDFILE
sleep 1
$APP & pid=$!
echo $pid > $PIDFILE
echo “Done.”
;;
esac
exit 0

I’ll have to make more of my monitoring tools run using init scripts and rely less on cron.

I was testing an upgrade for MySQL, from 5.0 to 5.1, to gain Row Based Replication instead of Statement based replication (as statements are a cause of SBR breaking, and it doesn’t proceed til I tell it to).

So, in my usual lazy, just do it, manner, installed MySQL Server 5.1, pointed the datadir at the old so that I’d have my data pretty quickly up and running.

And the installation worked, until I started MySQL.

I use hMailserver for email on my box, this uses MySQL for it’s settings and database storage (which is good actually, a config file is inflexible for multi domain environments).

The database used by hMailserver is InnoDB, most of my needs are well serviced with MyISAM.

In my rush, I didn’t do a backup. I didn’t copy the data folders, just went right over it, started, and got an error:

100610 22:15:32  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files…
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer…
100610 22:15:32  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 1204538731.
InnoDB: Doing recovery: scanned up to log sequence number 0 1204538731
InnoDB: Page directory corruption: supremum not pointed to

It was shut down normally – using the Windows Service, the issue is that the MySQL database won’t work with the log files from the previous database for some weird reason.

I’m still yet to determine if any recovery can be done on the log files, making them usable again, however I have doubts.

I reinstalled my mailserver and did a quick setup on it today to get us operational again. And learnt a lesson – always backup.

Actually, I should have had a database backup of recent, I don’t. I have no database backup – I only have the datafiles as they are.

That instance of MySQL is rarely used, it’s most usage would be the mailserver, followed shortly by the replication I get it to do from the server in Newcastle. It is my home windows test server, I also have a linux one that I planned similar for – except that does much, much more and data loss there would very much annoy me.

So, I ask myself – why is there no backup of it then? Because, there isn’t.

I MUST use this weekend (which thankfully is a long one), to:
A) Figure out if that log is recoverable, and move back to it, then onward to MySQL 5.1 using a mysqldump.
B) Upgrade To MySQL 5.1 for the windows and Linux box here, and the box in Newcastle.
C) Setup replication using Row Based Replication for those databases I have that need it.
D) Design a regular backup script which backs up the databases locally, something like weekly (With email reporting).
E) Update regularly. 5.0 is really old now.

Reconsider using hMailserver is also on the cards, it’s not it’s fault, in fact, it has the ability to use it’s own seperate instance of MySQL (and does by default, and that’s what I currently have in Newcastle).  But, I want the ability to access that data – later for user purposes, greylisting statistics and so forth.

UPDATE: I found the issue! It turns out the ibdata1 file I needed, was in C:\MySQL Datafiles. In the Program Files folder was a 10MB ibdata1 file, which is why the error messages were occurring.

The Database came right back to life the very moment I shut it down. Copied the file from C:\MySQL Datafiles to the version 5.0 program files folder, and started the version 5.0.45 MySQL server with that file, and the log files intact.

Remember to remove the innodb_force_recovery option if it was added, and it can help to change default-storage-engine=INNODB to default-storage-engine=MyISAM – so that the server starts with MyISAM instead of INNODB.

This will not work if you don’t have consistent logfiles and data files – but otherwise, it’s a save!

Now, the plan is to dump all that data using mysqldump, and then go forth and better my lazy processes for database server management.

The problem was cookies. In order to recognise the person posting, the site requires cookies, and that’s just something XHR can’t manage outright by itself. So, I devised a better strategy.

Using GM_setValue, I checked the location in the script to ensure it matches where I want the Cookies to be stored from, and set the value of MyCookie to document.cookie.

Then, when it comes time for a request to be sent, in the XHR, I add a header: Cookie: GM_getValue(“MyCookie”). This fetches the cookie from the script settings, and sets it as the cookie for the request.

Then, it was just a simple task of encoding my data, and sending the request.

This is of course a friendly reminder to ensure you only install scripts you trust, else you might find someone catching your cookie data and using it for undesired purposes.

My colocation box is a HP DL360, G3, so it’s not exactly bleeding edge, but it’s a fantastic server, does it’s job and does it reliably.

Last night, we had a planned outage for just a short time to move the box to a different power situation, this was done and the box came back online.

It dropped off 6.30pm, came back 6.45pm. Then, 7.15pm there was notices coming through – I’ve began to ignore the warnings for a while now, so I failed there but it wasn’t too bad, because..

.. My colo provider, Inticon detected the port was flapping 20 minutes later, and went back to have a look, at 8pm he had advised that the issue was with the processor fan assembly, only one fan was spinning – strange, it was OK for a while before hand.

He went out and about to find a replacement part, and tried to find alternative measures, but came up empty. Considering the nature of the part – it’s pretty server specific, so it was understandable that we had no alternatives.

I then began hunting around for the part – found one, $185 online. I thought, wow, this must be a pretty premium part..

The contact at the colo provider went even further, and actually advised me there was a box on eBay for $250 – no kidding.

I looked at it, considered it’s marginally more expensive, but gives us a lot more hardware to play with as well. I contacted the eBay seller, asked if he could accommodate pickup for tomorrow – he was OK with it, fantastic I thought..

So, I head to bed – we can’t pick it up at night, it’s just not THAT important (and I had started work 8am, so by that point I was very tired). I think about this for a while in bed, then somehow drift off to sleep.

One of the kids is sick, so they crawl into bed beside me and annoy me, so I get up at 6am, head to sleep on the lounge, peaceful.

The next morning, I get up, sort out a pending PC related issue for somebody (it was arranged already), then the seller was interstate for the morning, not back to 1pm, so that was as good as we could get.

He then contacted us, advising he wouldn’t be back til 3.30pm – considering we were getting ready to leave around then, that’d work out perfectly. We got near the address in the southern Sydney area, but he calls and advises 4.30 – he was delayed. No problem, can’t do much if he isn’t there..

So, 4.30pm, he arrives, we have a look at the box, it works, he upgrades the CPU and RAM and throws in a 15K 36GB SCSI drive – fantastic I thought, more spare parts. The box worked, so in the boot with it, time to hit the highway for Newcastle.

We hit Newcastle at 7pm (after litterally, racing down the freeway to Newcastle), and the colo provider contact was already there – fantastic.

I take the server into the colo room, upgrade the box with what good parts I could – extra 1GB of RAM, 3.06 to 3.2Ghz Xeon, swapped over the fan assembly.. Power it on, it takes some time to boot, unusual, it eventually does boot.

Then, I enter into windows, get a ping going, get task manager up – make sure the virtual machine kicks over, it does. Box stays up stable.

Pack up the changed parts into the ‘new’box, and get the colo provider to hold those incase something else fails down the track – pretty much a fully working machine with exception the front panel – which shouldn’t fail again.

Then, head back home approximately 30 minutes later, by this time the kid who is sick complains about pain, ends up needing a toilet break, all sorted.

Hit the highway to go back home, arrive at 9.30pm.

Wait for the colo provider contact to remove some trickery done to assist with the downtime, and I modify the SMTP route I put in place to redirect the affected domains to my router.

The outcome – all mail leaves, some spam shows, delete this, and I update the database, all in sync.. fantastic..

All up, nearly a full day, 460KM’s travelled, $14 spent on food at Aldi in Sydney, $22 spent at Maccas in Newcastle, $250 spent on a pretty darn good deal (compared to $185 for the processor fan assembly) – and we are all back and operational.

Hardware failures are REALLY annoying – especially when you can’t just call HP and say fix it in 4 hours.. – but we did pretty good for a 23 hour restore time (and still managed to keep some essential services up anyway).

And the dedication of Inticon is outstanding.

Had this been Servers Australia, I have no idea where we’d be. Part of me says ‘online sooner, cause they have HP DL360′s (or had them)’, the other says ‘getting Jared to get off his arse and take some damned action’. I’m leaning towards the latter, it’s just typical of previous experiences.

Note to self: work harder on high availability.

I use a realtime implementation to allow for easy addition of users and to have the data I want in MySQL (makes it easier to monitor activity in CDR, etc). The problem recently was as a result of an upgrade, there appears to be changing to Asterisk’s debug log. Previously, Asterisk debug output would end up in the full log.

This made it easy to review the traces of an issue being experienced, and follow them back to the source – the log would simply contain the full sequence of issues leading up to the ‘issue’, and what happened after.

The newer version of asterisk however doesn’t enable debugging unless the startup option ‘-d’is specified. safe_asterisk restarts asterisk in the event it bombs out – and one has to admit – crashing doesn’t happen often with asterisk, it’s ‘reliable’.

So, whilst I was reading the log, I was seeing messages such as:
” res_config_mysql.c: MySQL RealTime: Failed to query database. Check debug for more info.”

That’s a nice warning, it tells me to find the source of the issue, check into asterisk debug log. So we open the seperate file appointed to debug (it removes clutter), and it’s empty.

So, I look into the asterisk configuration file – is debug enabled? Of course it is. Spelt correctly? Yep. Where’s the debug – how can you debug a bug without debug?

I check Asterisk ‘logger show channels’– this is described to show logging channels.

Sure enough, there is a line that states my ‘debug’log is of “File” type, and is “Enabled” and is configured to be “Debug”. Asterisk isn’t doing anything wrong there.

Then I look into ‘help’– core set debug. Could it really be, asterisk requires the debug option to be set? It does. You can set it there, or alternatively, you can have it enabled on startup in using safe_asterisk.

In the safe_asterisk script (nano -w /usr/sbin/safe_asterisk) – find “ASTARGS” and edit it to include a “-d” – this enables asterisk debugging information to a log file, by default, asterisk doesn’t log debug information.

This is confusing. The problem turns out that real-time now has an additional field required – ‘regserver’, adding that field seems to have caused asterisk to stop spewing the error message.

The issue may be ‘better’managed by asterisk real time mysql having a seperate SQL based log for any errors it experiences. In a production system, you might not have so many errors.

Although, another point to note is the mysql realtime module is very chatty:
 res_config_mysql.c: MySQL RealTime: Connection okay.
res_config_mysql.c: MySQL RealTime: Retrieve SQL: SELECT * FROM sip_buddies WHERE name = ”
res_config_mysql.c: MySQL RealTime: Connection okay.
res_config_mysql.c: MySQL RealTime: Retrieve SQL: SELECT * FROM sip_buddies WHERE name = ” AND host = ‘dynamic’
res_config_mysql.c: MySQL RealTime: Connection okay.
This query might return no results, so it begs to question why is it even executed?

Enjoy!

Previously, you had to turn it off for peers stuck behind NAT, this is resolved in newer versions of Asterisk which attempt to determine if a peer is behind NAT, or if the peer is connectable.

The method is simple, for the peers affected or unsure, set the canreinvite option to ‘nonat’– this will cause the audio to be connected directly to the peer if possible, otherwise, it will packet2packet bridge the audio.

It is important to have the right mix of codecs on the server and specified in the ATA settings also – without it, you’ll just have a mess to sort out.

NAT is a big problem for VoIP – nearly every gateway / router will implement NAT differently from what I have been told. The BEST configuration is to have the ports forwarded. This isn’t hard for many people – it’s just a matter of finding what your router calls it, and then forward the ports. Once the ports are forwarded, there should be only codec mismatches to worry about – and of course, if your codecs are specified right – well, there is no problem – the audio should be connected directly, rather than bridged.

Asterisk 1.4 and 1.6 have had many advancements to SIP and RTP implementation. T.38 support is at a strong point now – not completely tested by me, however with T.38 support, it greatly reduces the errors you may experience trying to transmit fax over ulaw / alaw.

Check out the UPGRADE.txt for Asterisk 1.6 – the list is massive and impressive.
Enjoy!

One of the key features I took advantage of is the context option for each SIP peer. It is possible to exclude one group of users from calling another group by using contexts.

Let’s say I want user 300 to never be able to call two users, 400 and 420. 420 and 400 can call each other, and call 300. User 300 is simply someone who will get bullied by 400 & 420, but won’t be able to get back at them.

This is accomplished easily through the use of extensions and SIP peer contexts.

I set the context for 300 to ‘victim’. I set the context for 400 and 420 to ‘attack’.

Then, in extensions.conf I create the appropriate configurations.

[victim]
exten => _310X.,1,Dial(SIP/${EXTEN})
exten => _310X.,n,Hangup

[attack]
exten => _4X.,1,Dial(SIP/${EXTEN})
exten => _4X.,n,Hangup
exten => _3X.,n,Dial(SIP/${EXTEN})
exten => _3X.,n,Hangup

The sip configuration for each is similar to the below:
[310]
type=friend
username=310
secret=yamomma
allow=all
context=victim

[400]
type=friend
username=400
secret=md5
allow=all
context=attack

[420]
type=friend
username=420
secret=secret
allow=all
context=attack

Try as 310 might, there is no way they can call via the system to 400, however, 400 and 420 can ring 310 as often as they wish.

You can of course add the 4X route to the extensions, and playback a message to the user.

This method can be used to restrict calls to certain areas, such as 1900, or international numbers. It can also be used as a call control method, in the event you don’t want a specific extension calling someone.

It is important to have a peer and user definition. After upgrading to Asterisk 1.6.0 on my test server, I found my test development server (used for simulation) running Trixbox 1.1 (yep, ancient crap), would not accept calls.

I upgraded to 1.4.22 on the test development server, and found the same issue. This was resolved by changing the sip definitions used for interserver communication to be ‘friend’type, rather than ‘user’type that appears to have been automatically created by Trixbox – no longer used.

Oh well, back to enjoying a long weekend here in NSW. Another day for me to experiment some more and try and push forward with the updates planned for OzVoIPStatus!

Speaking of OzVoIPStatus – one of the areas I’m having difficulties enhancing is the method in which providers may specify plan data – some have included calls, others have none. Some have unlimited calls, and others have untimed call rates. It’s not so much a difficulty designing this – it’s more in the way of presentation on the page.

Enjoy!

Then, after a short week wait, a response. It’s not a disappointing reply, but it’s not successful just yet.

I made the short list for the second round of interviews, which being very soon now, to me looks like they have a tough decision, and they are looking to exclude candidates.

Knowledge is what allows us to strategically position ourselves to suit the requirements and acheieve the goal, in this case, to be successful for the application.

There is not a lot of knowledge available, we know the company, we know the current exploits on their website (well, maybe they don’t), we know of some of the bugs, we know of what the current users experience, and we know a little about the company structure.

What we don’t know is the other candidates – what do they have to choose between? That’s something I don’t know, which of course, makes selling yourself, a simpler process. They are doing the comparisons, I’ve simply got to demonstrate what they are looking for in a new employee, can be found with me.

The previous interview went for a good 45 minutes. I’ve set another hour aside for this interview, to allow plenty of time for a good solid discussion. I feel confident in my abilities to fulfill many of the tasks they are asking of the right candidate. I can administer servers and perform preventative maintenance, I can optimise code and ensure a positive user experience. I can contribute to discussions, and design to meet objectives. I can create new applications, and optimise what is already there.

But, I imagine so can a few others. The average pay rate for the position is around $55k per annum. I’ve gone for a figure lower than this, because it’s my ‘first’employed position in the area, and at the moment, I’m keen to get away from the support scene.

I’ve put a fair bit of thought into it, sure, I can support users day in, day out. But the frustration comes when it is something so simple: Have you actually read the quick start guide?

Or, when someone calls, for something that can be resolved by email and thought out in detail, rather than over the phone.

Or, when someone has an unsupported setup, and calls in expecting immediate support for such a setup. I don’t have 1 hour to spend on the phone helping you do the job of a technician, it’s not fair to those who have genuine issues that WE do need to resolve.

I don’t like phones, despite my interests in VoIP, telephones annoy me. They ring, they get answered, and you just think: “Who the f@#k dare disturb me answering this email.”. This of course means the person on the other end of the phone may not be greeted by the most happy sales person (luckily I don’t answer Sales enquiries anymore). I like to get to the chase, find out what the problem is, give them some troubleshooting, and tell them to go away, so I can assist those who thought patience was ideal, and used email.

Many of those issues just won’t be present in the new job – it’s for an online website, and one which doesn’t actually have a phone number – bonus. There isn’t a strong support aspect in the job, much of it is simply creating applications to meet design specifications, maintaining services, and improving the code base (from what I’ve been told isn’t the greatest).

The site itself employs a fantastic array of AJAX, and therefore shouldn’t be a load nightmare to manage. There are many database optimisations I can put in place, and I’ve identified a few pages I’d like to view the code of, and see how they are currently doing that.

I’m aware of a few security related issues on the server, and part of me thinks – If I tell them, they may see it as a negative, that I would be using it to simply get the job, on the other, they may think it is a positive – I’ve researched the site thoroughly and already found a few exploits. I’m going with the safer route at the moment, I very much want this job, so I’m going to try and sell my abilities to them, and simply not advise of what I am aware of. I’ll just remember to fix them when I start in the position.

Hopefully, the interview can be easily cleared, and I make the number 1 spot on the shortlist, and blow the rest of the candidates out of the water.

Still on the edge of my seat.

Much of the items are as I described several months ago, mindless paperwork describing something that you have done countless times already – but of course, in this day and age, certification is not always just a bit of paper.

From an employment perspective it can show someone’s desires, what they felt motivated enough to complete on their own. It can also highlight educational aspects – how well someone can learn.

The Diploma in IT for me is a bit of a resume filler – to sit alongside my Cert I, II, III & IV studies and effectively leaves me open to the Advanced Diploma, or to follow more specific studies.

The Cisco Certified Network Administrator course has been a course that has found itself in front of my eyes many times – not the course, but rather the term CCNA. I had not considered proceeding a full path in networking, mainly because I know what I want, and can always find and work out what I don’t.

My ideal career path is one which focuses on development – Client Support is a fair way off from that, but it’s more of a stop gap measure to keep everything going and of course, add to the resume.

I’m not entirely sure where I’ll find myself with development – be it web or system related – within the next 3 years. I’m not anticipating remaining in Client Support for that long, it’s boring. I’m a problem solver, sure. I like to resolve issues, but of course, exposure to new issues makes it all the more interesting.

Client Support lacks the aspects I enjoy, the ability to replicate an issue, and fix it. Nearly all the issues I get faced with are the same old, same old. Restart, and viola, it’s fixed, or adjust configuration setting, and it’s fixed. It’s uninteresting, boring.

A problem solving challenge would be a new issue, a new investigation, a new solution, rather than repeating the same troubleshooting with the next caller (dependant on issue).

I had/have plans to push forward with a development business, but that isn’t taking off as well as it could be. I’m not a good marketer – when I review other attempts made by me, I do realise that.

Marketing Reps are expensive, and not something that can be afforded from a startup, word of mouth first requires mouths to give word out of. A telephone directory ad would just create phone calls – I can’t stress enough how much I dislike a telephone (yes, despite my large usage of SIP / VoIP when developing OzVoIPStatus).

Thinking of marketing, I’m reminded of that ‘thatradiobloke.com’advertisement, which makes sense – create a catchy advert and entice the listener to review the website to get the full details. Websites of course offer a LOT of information, more than a 30 second radio ad can convey (not every case applies, stupid flash splash pages for example).

So, back to where I was – where do I see myself in proceeding with Development? I could continue to try and follow up on creating a development company, and provide solutions to problems faced by IT users. What can I gain from Client Support? Patience and tolerance for the telephone, and the slow paced users on the other end. Not something I want much anyway.

What will my Diploma get me? Not a lot at the moment I imagine, but the future can hold many oppourtunities, so it’s important to keep doors open and ensure I have the capabilities to be dynamic (and have a dynamic, but extensive skillset). IT is a dynamic marketplace as well, new technologies replacing old (and I’ve used Windows Vista an approximate total of 1 – 2 hours).

I enjoy solving problems with programs, and creating solutions to IT problems, it’s ‘fun’.

I have just completed an accounting tool that I have been pushing through for a friend recently to assist with online data.

It uses Quickbooks, and interfaces a lot of his data with that provided by customers – all in the lead up to something bigger to happen later on.

But, as I go through all the bug fixes, and complete all the features and see the application form from a simple blank line of code to what is now.. 1006 lines of code, which helps with performing the tasks required in a very efficient manner – it saves on manual entry, I feel somewhat rewarded from the effort that has gone into it.

Sometimes it isn’t money that is the reward, but rather, seeing the completion of a development. 

I enjoy developing, and that’s in the full sense of the term, from theoretical, through to the stages of development, information flow, and programming, and then UI design. It excites me, and then you find the feeling of accomplishment at the end, after you have pushed through all the issues, and made something from – nearly – nothing, and of course, it works. Naturally, no item of software is bug free – even Notepad has a bug, but development and testing show this is pretty well completed – I test a lot of code as I go, to ensure bugs are identified quickly. It slows development ‘slightly’, because if you get it right, then you are wasting marginal amounts of time testing to ensure perfection, but, then, if you later have to dig through the code to identify your bug, then  you’ll be in a different mindset and stuck reading through the code.

I guess that might somewhat apply to open source as well, when you have multiple developers, they’ll have to read through another developers code and wonder ‘what on earth was he thinking there’– naturally, typos, and logic errrors – or in the case of phones – train of thought interuptions can cause errors!

I’m still feeling happy with the completion of that, because it now signifies the possible start of the next stage, which is more web based, and I enjoy web development the most (yes, ignore OzVoIPStatus, it isn’t exactly the most developped website I have).

Oh, some – like Sydney – may be wondering where my views on Telstra, Optus, “Fox” Terria, the G9, and Broadband are. Currently, I’m avoiding those topics for a little bit, I have the new job to keep with, TAFE studies, and where I can fit items in, development – if only I could turn that into a full time, well paid job like my client support job.

Enjoy!

The accounting application used is one of the known brands of accounting applications – there’s only two in Aus, and it’s not MYOB.

The company who develops or on-sells this international application to Australian users has it’s international version SDKs – pretty much a requirement for developping targetted applications to interoperate with their software – the SDK for US, UK, CA, and Online are available freely, online. Register, download a file, install it, SDK access in your browser in just moments.

The Australian version however, doesn’t differ significantly from the US and UK versions, with one exception. Tax. There is a feature built into the application to allow for a transaction to be entered as amounts ‘include’tax, or, if not ticked / or in SDK terms ‘specified’– then it assumes it isn’t tax inclusive and adds tax.

So, I dug through the SDK, found a “IsTaxIncluded” option related to non US versions – sweet, plug that option in, run an import and we should have tax inclusive accounting – it’d make it presented better when they are ‘printed’.

That didn’t work, the result from my XML request was a rather standard reply, showing it successfully took my request, processed it, and in my request, I specified “IsTaxIncluded” with a true parameter.

In the result, I read through, saw that it had added tax, and it simply ignored my “IsTaxIncluded” parameter. I’ve looked at the SDK information for what I am importing, and if I was running the UK version or the CA version, this would probably work without question, tax included.

The problem with the Australian version is the company that distributes in Australia charges $1999 to register as a developer, and the API is available, for a fee of $199, or the SDK for $499.

Interesting, I want to develop a rather small application that will simply interoperate with this companies bulk, slow processing peice of junk – and they want to rip my wallet out for $500 to provide only SDK access so I can see if there is a feature that can do it. Of course, I’m not going to pay that to see if such a TaxIncluded feature is on offer in their new SDK. Get stuffed. Why would I pay for information that the parent or development company provides freely for all other international versions, and the Australian company are simply stifling the development options?

I understand businesses are in business to make money, but fair dinkum, this company really don’t want to encourage users to take up their offering, there are alternatives which simply don’t require outlays of funds to do such a simple task.

It’d be OK to charge for SDK if they exposed the object via a query for a specific item- so let’s say I pushed through a query with a “Describe” option, it would give me back XML with all possible properties that can be set for that specific transaction – but nope – empty.

Even more so, this same company offers an Idiot Interchange File format (IIF). The file format has to be one of the worst. It’d have been easier to offer XML as an IMPORT method rather than work on a file format which has many, many flaws and restrictions.

What seemed to be the case was that a cart system was purchased, and just on the fly someone decided to break away one of the existing payment modules and slap something together which would work with eWay.

The system was passed back to them in ‘apparently’working condition.

When I looked at it today, I was surprised that such sloppy coding existed.

I saw an SQL query formed, referencing a nonexistent array (as in, no array was created whatsoever on that page). This problem was producing 500 errors for customers, and whilst the system was happily charging the credit cards, the SQL query formed after it was broken, so orders weren’t getting marked as paid.

The array not existing at all, but being referenced suggested that they had ported code from a previous module, which did include the array, and simply didn’t error handle or even test this (an array which isn’t declared throws a types exception when it doesn’t exist).

So, the fix naturally was to adjust the SQL query to suit the situation and get that live, but I can see immediately that most of the cart is a mess from what I’ll call ‘chop coding’– rather than develop the solution properly, buy someone elses and chop and slice til it works and the customer pays up (princely too, $3000+ for crap).

Lately I’ve been keeping occupied with finishing a project. You know, each time I get near that “it’s gotta be done” point, I find there is a little more to go. Still need to draft up some good old content pages for this, and then get some more behind the scenes stuff done. Then I can probably call it “Done” and set it live.

I could have it done tomorrow, thinking of it tonight, but when you hit that favourite stage, known as testing, you generally do find something needs to be fixed up and corrected so it performs nicely. It’s all part of the fun of creation though. Look at it, if it’s still not right, go back and fix it, repeat until satisfied.

I’m sort of excited to some extent. I want to see how it will go, but, I also want to make sure it’s done right, and done well to ensure that we don’t have any hiccups.

It’s taken a bit of time, but it is enjoyable at the same time. It’s something I can enjoy putting together, and something that will work well.

Enjoy!

I think the better solution to this issue (and issues related to SMTP auth attacks, and 550 errors), are having an error counter in the mail server software, and if they hit a number of errors, the connection is then blocked and the IP is firewalled (denied a connection) any further.

This is rather easy to implement, because most servers can choose whether they accept or deny a connection, and in the case I am thinking, it should be a matter of looking in an array, seeing if the address is firewalled, if it is, then the connection attempt simply isn’t allowed and the server doesn’t open it at all.

That, will in turn make spam attempts like the 550 ‘guess’ineffective, and ultimately destroy the spam attempts, getting closer to eliminating the needless waste of bandwidth the attacks on mail servers.

Spammers who try methods such as address guessing will find themselves unable to connect to many servers, and could perhaps consider a new career, perhaps as garbage collection, to repay for all the garbage they spit out.

As it stands, it is possible to analyse an army of logs I have now, and calculate the spammer IPs and place those in a database, which will be loaded by the server on startup and stored in an array.

The server then on connect attempt, will search that array for the address, and if it finds that it is in that list, then connections are denied.

Adminstrators of the server can then go and check a list of addresses and see if any are of interest or not, and if necessary remove them.

Most good configured mail servers which find they are disconnected will retry for days after, so reviewing a list of addresses (sorted by date added) should be a painless task to do.

This feature would be used in conjunction with greylisting for example, which already tells senders to try again later.

The IPs banned would be those which score 5 errors in a single connection, as these are typically the problem connections, from 221 and 89 IP ranges (but I’m not racist, or is it ‘IP-ist’?), so won’t block full countries of IPs.

This isn’t a complete solution to the identified issue, but, it’s a start at making it even less effective.

Enjoy!

I enrolled into the course then, and have found that I have a total of 800 hours (in TAFE terms) of study to complete to get my Diploma in IT.

The hours are a little packed up, but those 800 hours are spread over around 23 seperate modules, and many of them aren’t really complex. With names such as “Design A Server”, and “Configure an Internet Gateway”, you can easily see that it isn’t really a packed up task.

When I started IT courses, during a slow period here on the Central Coast, I decided that I would study for a while, and see what doors opened later on.

I completed as far as the campus I was studying at was prepared for, of my Certificate III in IT, and despite not completing it 100%, completed a great deal of the course and then enrolled into a Cert IV afterwards with a different arena.

Throughout my studies of Certificate III, it was constantly drummed into us, that IT isn’t actually working with computers.

IT, is in fact reporting on technology. And a lot of reports are generated for some simple things.

Build a network for example, and the requirement is several pages of text detailing what the network should be capable of, and what expectations and maintenance requirements it will have.

And, of course, what business will gain from the network.

You look at that and you think, what a big waste of time, reporting on what essentially are obvious recommendations obtained through observation and benchmarking the network in place.

None the less, the concept of IT is writing reports on what will happen if this plug is pulled, and what will happen if this much money is spent.

Does management actually read such reports with interest? Or do they simply call the IT Guru in and stare amazingly at his wisdom that the business has yet again been able to produce a pretty pie graph on a wall?

Reporting isn’t that bad though, it’s always a constant look at what technology is doing for a business, and keeps everyone on top of the technology. A new CPU is released, the average professional would buy the CPU, turn the machine off, pull the old CPU out, plug the new one in, turn it on and see what magic they can do.

In Business, this is changed dramatically, prior to purchasing the CPU, a report is written on what the current system is capable of, what the new CPU will acheive, what performance benefits will be delivered, what sort of service effects it will have, what it will cost, and what enhancements can come in the future.

Then, if that’s approved, you would buy the CPU either using a purchase order which might involve a few emails and calls to the ‘Financing’department, then,  you would get the CPU.

With the CPU in hand, you must then, write up an implementation report, and get the implementation approved by management, to make sure they are fully aware of what is happening, what will happen if something goes bang, what if the CPU does not work and is faulty, essentially covering the details from start to finish.

Then, with that implemenation report approved, you can actually proceed with the implementation. Essentially, wait til after work hours, turn the server off, remove it’s cover, take off the fan, remove the CPU, install the new CPU, add a bit of cooling paste, install the fan, place the cover back on the server, power it on, and test to ensure there is no problems, and also test to see if the claimed performance gain is realised.

Then, yep, there’s more. You would write up a post implementation report, stating whether implementation was effective, what gains have been realised, and if any thing should be documentated about the installation.

And that’s how you install a CPU into a server in a real business environment, according to my Cert III studies. I have some doubts this is somewhat realistic, because I just don’t think business would pay someone to write 3 reports for 15 minutes work, but, it could happen.

The diploma is more a theory course then anything from what I gather, as it is a lot about how you would approach a situation, how you would design a server for example, or how you would ensure client privacy (yep, that’s one of them).

Not only do you “Design A Server”, but you also “Build and Configure a server”, and “Build a security shield for a network”.

From my previous studies, a lot of the IT Systems Admin arena is focussed on Active Directory networks, setting up Windows Domains, monitoring them, making sure they tick like clockwork, and upgrading them when required, securing them.

Writing reports isn’t really a big issue with IT, it’s great to some extent, as you gather a lot of data in a report, that can clear the water a little more, and you by research (required for many items), you naturally stay on top of the latest technology. You probably shouldn’t have a personal preference for any specific item, so that you can compare both side by side and always pick the best item, and know why you are picking it – something not always common with the ‘gamer’style CPU purchase, where they might buy a CPU because it has 4 cores, of which 2 are actually used.

It nearly all focuses on networking, ranging from managing a project, to gathering data for business, and translating business needs into technical needs, and after that, it’s maintaining system security, securing a network, ensuring user privacy, and so forth

Interesting, but not entertaining.

I estimate that if I actually got stuck in every day for a solid day, the course could reasonably be completed within 6 months.

I don’t get stuck in every day of course, because I simply just don’t get motivated much to get stuck into it (or anything else really). It’s sort of annoying that I can’t get buzzed into completing items like I used to be able to, it’s just so hard to get started on something, but, once I start something, I can generally follow it through to finish.

The start is the hardest, once started, the finish is nearly always insight. This same situation applies to OzVoIPStatus for example, where I have started a new layout, new design, and new pages for the site, and they are there waiting completion. One of the new pages has been sitting there for a while, waiting for me to code the rest of the “Provider URL” line that I had started weeks ago. It’s just “HARD” to get started.

TAFE work is easier, because it’s new and unstarted, once I start something though, and walk away from it, coming back to it another date isn’t interesting, but I do get there.

Maybe this is related to my arrangement of the room, or maybe something else, I’m not much for Feng Shui for example, I’m not 100% sure what’s dropped me off the regular motivation that I used to have to dig into and complete things, like I used to! I’m sure it’s something that can be sorted though by finding out what makes me comfortable when I was able to get stuck into things.

Maybe it’s just that firefox window with Whirlpool open that’s distracting me from what I’d usually do.. hard to tell.

Networking, gotta love it.

Enjoy!

So, tomorrow is going to be spent tracking down the right part, and then seeing if they can supply, if not, venturing a little further south to find one!

The auto electrician backed up my thinking that $75 was a bit rich to pay for it, but, they were cheaper then the other mob, who were both deceitful and rude.

Hopefully, they’ll be able to exchange it, if not, then, they can probably go find us rear spoiler LED brake lights.. I think that’d be pretty cool to get working again.

Anyway, more time and money spent fixing a window that should really have been tested before they passed the pink slip when it was sold. Bugger.

The other issue, was the speakers in the back need connecting properly, I’m able to do that, we just thought of a better way. Tape the new wires to the old, pull the old through, and remove the old, leaving the new one in the right place! Fantastic idea, hopefully it works in practice (so as long as the wires aren’t anchored anywhere).

In other news (sorry Sydney, not going to do the FTTH vs FTTN scenario tonight), I was fixing a laptop today, and discovered it had a rootkit on it.

You won’t generally know a laptop has a rootkit when you see it first off, mainly because it looks, acts and feels like normal most of the time, that is, until the ISP disconnects the dial up connection obviously due to SMTP activity (great idea this, if a customer is sending out spam – easily tracked, then chop the connection, repeat until they leave or its fixed).

The rootkit was a difficult issue to find and solve, mainly because it really didn’t seem to have many problems in testing, and all did look good, but I did indeed check for viruses, and PC-Cillin missed it entirely!

So, I had a poke at netstat -bno (this lists the binary that started it, numerical IPs to save on dns resolution, and the other option shows process ID as well).

Anyway, after looking at that, it was very clear the system itself was trying to spawn SMTP connections, and this was all from services.exe, so not something you can just “End Task” on.

The local computer shop wanted to charge the young girl $300 to fix the issue. Well, local computer shop, you better employ better technicians, because that issue cost no where near $300 to fix.

It was complex, it was a little out of the ordinary, but by no means did, well, we call technician time at .. $33 an hour for instore? So, 10 hours of work. No where near that. I had it for perhaps 2 hours or so, and it runs fine. No strange connections, no more foreign process running, all looks good.

The intention behind the rootkit was to find and collect mail servers, and package these and send to the admin of the IRC based bot net, so that they could use the network to spam.

The idea further was to go in undetected, so that they could escape detection (and it pretty much did escape every regular test I’d have done to find it, except netstat, which showed SMTP connections, and trend bitching about scanning outbound mail).

Trend failed to detect it as well, which is strange, the activity seemed pretty regular for an SMTP virus. Generally opens multiple connections to different servers in a short timeframe.

A regular PC user generally only opens one or two SMTP connections to one or two servers.

The fact this was trying to talk to 10 or so (I have a router I can block access to, and did so), mail servers alone was a big hint to perhaps it being virus activity.

Anyway, that’s why they call it windows. The both versions of windows can be annoying, and a little difficult to fix.

Enjoy!

I don’t have much involvement in it, it’s just running there, my partner does more with it then I do.

The trivia room runs off a MySQL database (primarily due to a redesign decision to give it more uumph then flat files), and I was just looking in their on an ‘occassional’view, and noticed that the answers were coming through quickly, but the messages that follow, that tell the user the score, etc, were taking seconds to come through, possibly frustrating players with the extended delay that follows (ie waiting on a new question).

Anyway, I was pondering whether it was the query causing the high load, or whether it was something external to that, or something else (when it comes to databases, the issues are either bad queries or bad design).

The query was an extended union all query, which I thought wasn’t going to do much benefit like that, so I changed the code base to instead execute single queries to see if that was the performance issue (its more work to union results together then just fetch more).

It didn’t work as planned and I was then thinking, wait, I wonder if this table has any indexes or keys. Turns out it doesn’t.

So, I added indexes on username, username and date, and date and id fields on the database.

Amazingly, after it took its time saving those, the performance when from slug, to super fast in seconds.

So, an index is important, as your database grows to millions of rows, you need to have an index for high speed data access (and it really does give you high speed data access).

How indexes work ? Essentially they are small files of data pointing to full row record positions, so that way when data is needed to be found, less scanning of the full table is needed (just go ask the index).

Setting them up depends on what you use in your question. I used username and date fields because they are predominantly what the where queries are based off, we ask where are all the fields with this username, or where are all the fields with this username and date.

It’s a lot quicker to fetch that way, because we now know all the positions with that information, so its just fetching, and then getting the data together into a recordset.

The server load should be better off now too due to that issue causing it to see MySQL slug down to a crawl to fetch points for a specific user.

In unrelated news, it seems like we might be stuck with our unstable ADSL2+ service, simply because the Optus DSL ports have restrictions on the profiles they allow to be set, I’m told a 1500/256k service is possible, as is a 8192/384 service, and finally, an ADSL2+ service with a limit of 20,000 / 820 is possible.

So, we are having drop outs primarily due to our attenuation being too low then the service can handle reliably, continuously, so, the fix is to put a profile somewhere in the middle on, something that delivers us around 8Mbit with 820kbps up.

Unfortunately, the Optus ADSL1 limited profile is 8192/384- exactly the crap I had from Telstra Wholesale, and the reason I moved away (faster upload speed, cheaper service, no STUPID artificial limits).

Anyway, I already know that the 8192/384 service was stable on my line, and free of drop outs, and if I was going to get the same on the Optus port, then we might as well as just go back to ADSL2+ mode and see what else can be done for the drop outs.

It seems however, we are stuck with them, they aren’t really much of a problem anyway, except if they happen during the day on a VoIP call. I don’t care if I’m asleep and it drops out for one minute and comes back, doesn’t really affect much.

I would rather a stable service, but, it seems, we can’t move address due to the 12 month contract on the DSL service, but also, it’s just recently emerging that the exchange is all out of ports (wow, they filled up fast).

So, hopefully, over the next 12 months, something happens to make our service stable (ie. technician fixes a fault, something gets done to fix the drop out situation if its crosstalk at the exchange, etc), or, we simply work around it, and see what alternatives pop up after 12 months (ie. Change address for stable service, or, change internet service if new technology emerges that provides equivilent benefits).

I wonder how many ports Optus do deploy in their roll outs compared to customers they have. It can’t be too many, because they do seem to fill up quickly.

Our ADSL1 contract wasn’t supposed to be up til February, I was pondering waiting the 2 months for it to finish, but concluded this situation might happen, and watching neighbouring Gosford, I was right, they filled fast, and no new ports have been added to cause Gosford to become available.

I can work with drop outs, they don’t affect us much at night anyway (though they do happen at 9pm as well).

I’ll see how the second line works out too, which is going to get a ADSL2+ modem eventually too!

Enjoy!

They don’t seem to have much in the way of Adwords integration with Analytics for testing a complete visit from click on search engine through to complete sale, and even then, they don’t report adequately on the path a sale took, or how long the sale spent on the different pages.

A more useful report would be one that could categorise the conversion failures, so that they are easily identified and able to be worked against.

Or, if that’s anti-Google (and therefore Google doing evil) then perhaps an alternative is ideal, a report that takes out all non converting traffic, so you can see the detailed sales statistics of where a sale originated, and how it finished (did they read everything, or just 10 seconds?).

Those reports could be used by e-marketers to determine what keywords work best, what doesn’t work at all, and what is ideal for use on a regular basis, and what has short term good or bad effects.

Essentially, the tool is a great tool for analyising traffic, but it doesn’t meet the standard required for isolating traffic which isn’t sales related, meaning the data is inflated, and doesn’t isolated clicks wasted on Google Adwords (where the viewer didn’t actually view content, or where they didn’t wait for page load).

The reason behind this is, 560 clicks registered in Google Adwords, yet only 455 visits from that keyword registered in Google Analytics place the statistics in a question of accuracy (the numbers should match, right?).

I’d like to see someway of them improving the accuracy of statistics so that there is accuracy, and where a click on an ad doesn’t hit the site, credit applied for such cases (they obviously have the ability to detect such activity if you have Analytics running).

Enjoy!

Some one paying thousands of dollars for a ‘web site’and getting what amounts to automatically generated rubbish, or unplanned scripting errors and the like.

The business today I spoke to had an issue with a website they paid $3,500 for, and over the course of a few years, apparently near $5000. It’s nothing too flash, a bit of a flash entry page, then a template CMS overlayed with CSS, and a cart system added from an outsourced company bundled together for the sum of $3,500.

Amazing, right?

Well, the developer of the website refuses to support it any more, and the issue presented to me was the check out process was broken, and taking credit cards wasn’t working.

So, I got into action later on, working from the ground through the process, isolating the error, and then finding a fix for it.

The first issue was a lack of recognition of currency, the cart thought “AU”, where as the actual is “AUD” (matched in database).

So, I overrode its detection and coded it for AUD, and that works for that bit.

Fixed in moments.

Then, came the second issue, eWay payments not being processed. Not a tricky one, considering the error message told us all anyway. The system uses an XML sheet POST / Response request to send and receive data from eWay, and it was not able to resolve www.eway.com.au . No biggy, that’s easily solved with coding the IP into the system instead.

Well, I was wrong, the SSL certificate needs to match the domain name for XML HTTP, so that meant the hard way about it, getting DNS resolution working on the Intaserve web hosting.

I contacted the company and advised DNS resolution wasn’t working on one of the servers they have, and the response I got back consisted of a ping attempt to the computer name of the server.

Gee, I didn’t say I couldn’t resolve it, I said IT couldn’t resolve.

Anyway, I tested moments later after sending them an email that they needed to investigate DNS resolution on the server itself, and after testing, discovered the issue was fixed (probably from the client contacting the host directly).

So, that was all well and good. Fixed. Website works A-OK.

The question remains however, will I get paid my fee for fixing that? They claim they shouldn’t pay, because they’ve paid $3,500 to someone to fix it.

By pure co-involvement with the developer (not developing however), he believes that he shouldn’t pay as he paid for it when the site was established.

My counter argument here is that I didn’t develop the site, I merely suggested I fix the website, and I get paid if I fix it.

The site is fixed, hey, I thought they’d have no issue paying my minor suggested fee, and I’m a productive person anyway, I use my time wisely, while they are considering whether to pay me, I have the job done, they pay.

I’m not destructive either, so, if they don’t pay, essentially it’s just $33.00 lost (my fee for fixing it, took about 30 minutes to an hour).

I’m still amazed at the $3,500 charged, for something with such an issue (it’s an inconsistency issue), and the other issue being a DNS issue.

But, I’m now amazed at the businesses apparent refusal to cough up $33.00 for the time I spent on fixing the website, which they claimed they so badly needed (even resorting to threatening the co-associate with legal action, he doesn’t actually develop any more).

It’s $33.00. If you are losing as many sales as you claim you are, the $33.00 will be easily found in the profits of future sales.

It’s not really bothering me, and I won’t name names here (except the webhost, they should have their own gear in order and make sure DNS resolution is possible for customers by having multiple DNS servers), it’s not really a big issue, it’s just amazing that they spent so much on what amounts to around half that in actual presentable work, and the reluctance to pay for something they were claiming cost them sales, and were ready to launch legal action to try and solve anyway.

I should up my development prices, if there are suckers out there paying so much for so little, I should start profiting. It’s in my best interests to do so.

Enjoy!

The fix was to simply run resize2fs with it mounted in read/write mode, I kept assuming it’d need them not mounted, so was trying to resize it, when it wasn’t existing.

I got some sweet speeds on the Exetel mirror today too, leeching CentOS 5 ISO (650MB) in just 12Mins  (around 998KB/sec).

The line drop out situation isn’t improving however, and I’m not certain I want to drop the speed profile down to the ADSL2 or ADSL1 mode for the sake of making the connection stay up more. A drop out every 24 hours or so is OK, sure, its better to have none, but, well, that’s not gonna happen sometime soon, here’s why.

A news article today reveals Telstra technicians have resorted to taping cables together with plastic bags to keep them dry, due to staff shortages resulting from the sacking of 12,000 Telstra employees (says the union).

So, what will happen when the plastic bags cop a good dose of lightning, and end up welded to the copper wire (or welding copper wires together). Will Telstra accept responsibility for the crossed wires and pay compensation due to breach of privacy on some mass scale (all those copper wires being welded together by lightning would result in much a case of people listening to other peoples conversations).

Who knows…  What will be the result of all this plastic bag repair work? What happens when the destruction comes too much to bear and the copper cables start requiring full replacement?

Or, is this some elaborate plan to derail the PSTN network to make FTTN a near requirement?

I found the news story amazing, at the same time disturbing, shouldn’t they have their cabling licenses removed for doing unsatisfactory work (surely, plastic bag repairs aren’t AUSTEL certified).

It’s a good thing we don’t live in Sydney though, where most of this seems to be the problem!

As per usual though, it pays to keep a second service (mobile phone) on standby incase of phone line issues, etc.

The Virtual Server issue being fixed meant that I could proceed to getting the next task on the list done,  updating it, which went through nicely, and allowed me to do something else, get asterisk running on it – yes, probably heaps quicker to setup a second VM, but I wanted one machine for many reasons, server load is one.

Our juice bill came today, and it was a good result, with the bill dropping on average 4KWH/day.

We just removed our off peak hot water systems equivilent load from the bill, and after todays measurement, looks like it could be dipping down some more with the new washing machine in place (larger washer, but less loads needed to meet demand).

I’m happy with the results, and hopefully we can see a natural progression in the drop for another bill, before we might either dip to the bottom, or find new ways to reduce consumption even further.

It’s good at least, that we have removed the equivilent of a hot water system from our bill. For the savings, we could be running 2 split system air conditioners (or, keep the savings).

Enjoy!

Homer: The right way, the wrong way, and the Max Power way.
Bart: Isn’t that just the wrong way, but faster.
Homer: Yes.

I’m reminded of that episode of The Simpsons each time I hear of someone trying to do something where something else is already available to do the task at hand rather simply, or where they could have done it quicker if they thought things out differently.

I generally try and find the best possible solution for any issues I come across, I consider the quickest approach, the effects of that, the cost (if any), and whether alternatives exist that will promote future technology choices.

The balance of these generally means the decision made is closest to the right decision where possible (of course, I did myself make wrong decisions, one of that was the layout of OzVoIPStatus, not the easiest for maintenance).

But, I’m deeply amused at someone taking the HTTP protocol and using sockets coding to simply use sockets and clean up all the HTTP request data on a program they are writing.

I would almost certainly have chosen to use the IE COM object, or fetched the URL using a HTTP Request and then took advantage of the DOM object.

Or, even, using XML for the request and response.

The amusing part focuses on the fact they chose to talk to the application using sockets, which means flushing out all the server crap, and what happens when they change HTTP server? Or they add an extra application that adds more header lines?

The application could easily break and would require more maintenance, then it would if a standard was used.

Funny, sure, but it’s a little backward and counter productive when you look at the alternatives.

More likely, they weren’t aware of the alternatives, and are great at sorting problems out in a different way, rather then finding a solution to the problem, they make a new solution to a problem best solved with different technology.

“A different approach”.

Enjoy!

Computers have become very much so beefed up that the grunt they have spends a lot of time idle, and only peak loads when you are doing intensive tasks, even then, they generally keep within 50% of the processors full capacity.

Game play will use more of that processor usage, but that doesn’t really come into question in the server environment.

So, just about many of the servers out there are very capable of having additional virtual servers added to them, and with the price of disk space falling, virtualization technologies are very much viable.

I run Virtual Server here at my home, and it hosts my linux test environment, this is to save me booting between OS and also allows me to have computing resources available to me simultaneously.

So, I don’t need a second server running, I don’t need to pay for extra hardware, power, or anything like that, when I want to run the linux machine (and we use it often enough anyway).

One common issue with Virtual Server Linux Virtual Machines is the clock being out of sync and audio style resources skipping ahead or behind in the established virtual machine.

The fix to that is a combination of setting ‘clock=pit’as a boot parameter, and also installing virtual machine additions.

If you run a CentOS distro though, installing Virtual Machine additions as recommended doesn’t work as expected.

The reason for this is, the ./install.sh script looks for a string in the /etc/redhat-release file.

The string it is looking for is the third one. So, on Red Hat (where CentOS is derived) it would be “release” - Red Hat Release.

On CentOS however, it is “final” – CentOS release final.

The fix to this? Couldn’t be simpler.

nano into /etc/redhat-release, and between CentOS, add a space, so it might read Cent OS release.

Make sure you have the kernel sources (yum install kernel-devel), and that will allow the virtual machine additions to install, and once installed, leave as is, or just change it back.

Use Virtual Machines, there really isn’t much going against them, they save power, they allow you to have the extra resource available (ie. Linux system) as well as your Windows machine.

And, Virtual Server is one of Microsoft’s good releases, it’s a great setup for virtualization, nothing silly like VMWare, just straight forward virtual machines.

Enjoy!