Home, 2024 2023 2022 2021 2020 2019 2018 2017 2016 2015 2014 2013 2011 2010 2009 2008 2007 2006 2005

Haddow Internet Network Status History For 2012

05:30 15-Dec-2012 Server3 back on line.
03:80 15-Dec-2012 Server3 down, no power to site.
09:50 11-Dec-2012 Server2 attack has been stoped, while the attack was hapening all main email was falling back to the backup MX servers, the attack was a direct one, but the way the network is configed it could handle it & had blocked all IP's that was causing the problem. It does take time to do this but only mail to this server was slowed during the attack.
09:38 11-Dec-2012 Server2 is having an spam attack from an outside network.

15:30 01-Nov-2012 Server2 backup link is up and running, dns and some other settings have been done..

14:00 26-Oct-2012 Update, We now doing up a new contract with another provider for the backup link and should have the link up and running with in a few days.
12:00 24-Oct-2012 Update, We had some nice words with Optus and found out they are now blocking all traffic originating from the Internet to there 3G, but all data originating from 3G out and back is working fine, there system is also assigning ip address randomly for short times. This is only on the Optus internet LNS & not other providers LNS. We are now looking into another provider for the backup link.
09:00 20-Oct-2012 Update, the ipaddress on the backup link keeps changing every 3 to 5 min, but also the IP it is reporting to be is not the IP it is, Optus knows there is a problem on there 3G and are looking into it..
19:00 19-Oct-2012 We are seeing some problems with Server2 backup link, the link is up but no traphic from outside can be seen, we are looking into the problem.

14:45 13-Sep-2012 Server1 backup online, we have had a report that the core router has stop working, it has be replaced with a backup one for now.
14:30 13-Sep-2012 Server1 link down, we are looking into the problem.
15:00 07-Sep-2012 Server1 mains interruption, The data center we use will be performing some work on their power systems. They do not expect this to affect any servers. Start time 0001 CST to 0400 CST.
13:00 04-Sep-2012 Server2 Software update, We had BIG problems, a reboot happend at 21:15, BUT the server did not restart, and even turning it back to the old kernel did not work, but we found the problem, it was one of the Hdd did not have a clean file system, so took it off line till the OS was up and running, once cleaned it too will come back online. Down time was 21:15 to 22:35.
18:00 04-Sep-2012 Server2 Software update, Since they did not do Server1 today, we are going to upgrade to the kernel on server2, it will have to be rebooted once done but down time should be only about 5 min.
16:00 04-Sep-2012 Server1 Software update, Due to problems on another server that was beening updated the same time they have helled back the update till Wed 5 Sep at 0300UTC time.
13:00 04-Sep-2012 Server1 Software update, We have been told by the provider that the main server will be having software updates, this will start at 0300 UTC and the server will not be accesseble for about 60min, this will mean no MAIN "emails, web, DNS" access, but backup server are in place.

10:30 31-Aug-2012 Server2 main link back online, modem crashed and the only way to fix the problem is power cycle the modem, no main problems was seen, just some support email were held till the link came back.
08:35 29-Aug-2012 Server2 main link down, due to staff away till friday it may not come back till then, but may just start by its self, but don't hold your breath.
10:20 20-Aug-2012 Server2 mains interruption, The meter change started early, at 09:45 but all went off with out a problem, and finished at 10:15.
10:30 20-Aug-2012 Server2 mains interruption again, this time to do a meter change on site, site will be running on backup power till compleated.
13:30 07-Aug-2012 Server2 main interruption, did not happen, contacted the mains supplyer and only the controler know that it had been counselled.
00:01 07-Aug-2012 Server2 mains interruption from 9am to 12:30pm (Weather Permitting), work on the over head wires in the street are being worked on during this time AGAIN, site will be running on generator backup duing this time.
08:30 04-Aug-2012 Server1 "Corrective Maintenance on UPS unit" went by with no problems seen from any body.
10:30 03-Aug-2012 Server1 UPS back will be down for "Corrective Maintenance on UPS unit" from 3pm to 6pm, all loads from this ups will be transferred to generator during this time.
20:40 02-Aug-2012 Server2 crashed, this was caused by a staff member knocking the wrong cable, The server was down for 10min, Loads on the server may be up while it does some rebuilds of HDD.
09:00 01-Aug-2012 There was a problem with some emails trying to get out, Receivers servers that was using junkmailfilter.com to filter there emails, and was bouncing the emails back, clamming they were virus spam. We have traced down the problem (one Receiver mark the email as virus spam, in stead of ok, so it blocked all H.I. mail servers) We have fixed the problem for now..

13:30 02-Jul-2012 Server2 mains back on line.
10:30 02-Jul-2012 Server2 mains down, now running on generator. They started late.
00:01 02-Jul-2012 Server2 mains interruption from 9am to 12:30pm (Weather Permitting), work on the over head wires in the street are being worked on during this time, site will be running on generator backup duing this time.

12:00 12-Jun-2012 Server2 mains upgrade compleate, there was no down time, the Office and server now runs on solar & mains, with generator backup.
09:00 12-Jun-2012 server2 mains will be down today for about 1hr from about 10am, for mains upgrade, the server will have no down time due to UPS and generator backup..
10:00 10-Jun-2012 We are going to be doing some power upgrades on site of server2, on the 12 Jun 2012, Time is unknow when.

06:20 22-May-2012 Link to server2 back online, there was a scheduled up grade at the whole sale Data centre that was not ment to start till 6am but for some reasion they started 1hr before it.
04:55 22-May-2012 Link to server2 is down, we are looking into the problem.

07:30 25-Apr-2012 There was an inbound DDOS attack that was affecting the link. The targeted IP has been null routed and network has returned to normal
07:05 25-Apr-2012 Main link to server1 keeps going up and down, we are looking into the problem
15:00 03-Apr-2012 During a generator test on site at Server2, one of the UPS failed, we traced it down to the batteries in the ups, this ups only runs the office not the servers, so there was no down time on the server, just some of the office equipment. We have replaced the Batts and the ups is back online.
00:00 01-Apr-2012 Daylight savings ends at 0300aedst the clocks go back to 0200aest.
Update: all server switched over the AEST with no problems

11:00 30-Mar-2012 Over the last week we have seen some emails are slow to receive, we have traced it down to NS4 not responding to DNS request for the main domain, so for the time being we have removed it from the registry till it is fixed.
08:00 25-Mar-2012 Server1 data server center is doing a load test on there equipment on the 29-03-2012 during 1600aedst to 1730aedst, the server and links should not have any down time during this work.
09:00 21-Mar-2012 Server1 data server center is doing an upgrade to the A.T.S. (automatic transfer switch) on one of the main power supplyes on the 22-03-2012 during 1800aedst to 1830aedst, the server and links should not have any down time duining this work.  
11:45 16-Mar-2012 Server2 back on line, still no report onto why it happen, but all adsl's via nsw with this upstream provider was down also there 3G data network.
09:00 16-Mar-2012 Server2 update, server is up but link down, support are looking into the problem, access to some websites, main dns, Office VOIP's and some other serveses are down, but the backup DNS and Main Emails are still working.
08:30 16-Mar-2012 Main link to server2 is down, we are looking into why.
14:00 02-Mar-2012 We had a mail attack on the server from 5am, it took a while to track down the IP's that were doing it and block them, some people might have some problems sending emails to yahoo now too, it should fix ist self soon as yahoo had blocked emails from H.I. Servers due to the spaming problems, but will unblock in a few days.
14:00 02-Mar-2012 Some changes were made on the 28 Feb to some dns settings, but due to a full stop in the wrong place, the dns server stop doing request for the domain haddow.com.au, this in turned took down 3 other servers, as it takes a while to find out that it did ont work just over the 4 days, but all is fixed now.
Some emails may be slow to come in this was due to the problem, but should still send ASAP.

14:35 23-Feb-2012 Server2 Link back on line, there is a problem upstream that is blocking access to Server 2 & 3 links, the link may go back down again to fix some routing stuff when the router comes back online.
13:30 23-Feb-2012 Server2 link down, Looking into the problem.
15:40 11-Feb-2012 Server1 back online, server was only down for 2 min for force the changes to be permanent.
15:30 11-Feb-2012 Server1 is going to be rebooted again in 5 min.
07:25 11-Feb-2012 Server1 back online, the problems with the IO requierd the server to be rebooted but the server did not come back straight away, there was still some change of settings that fixed the problem.
06:50 11-Feb-2012 Server1 is not responding, We are looking into the problem.
10:30 09-Feb-2012 Server1 is reporting hi IO we have be told by the provider that they are doing upgrades to the server, but did not tell us before they even started, they clame it might take up to 48hr to clear the problem, but we are pestering them to fix it before then.
10:00 08-Feb-2012 The data center (where Server1 is,) are going to be doing work on their core network infrastructure on 11th Feb 2012 from 1600 AEDST till 2200 AEDST. During this maintenance window datacenter staff will be performing software upgrades and hardware maintenance on their network. This includes rebooting a router to complete a software upgrade. There will be a momentary loss of connectivity (estimated 5-10 minutes) while reboot occurs.
11:15 03-Feb-2012 Server1 back to normal. Support needed to do a hard reboot to fix a locked file problem.
11:15 03-Feb-2012 Server1 update, the server is now not responding to any commands, we have traced down what had caused the main problem, but now can not fix that till I can get the server to reboot.
09:30 03-Feb-2012 Server1 has some problems, we are trying to track down the problem, but for now it may go up and down till fixed.

13:45 21-Jan-2012 Server1 back to normal for now.
13:15 21-Jan-2012 Server1 update, the problem is back again.
12:30 21-Jan-2012 Server1 update, there problem is up and down, we are still looking into what or where the problem is.
11:30 21-Jan-2012 Server1 is having a problem, Support are looking into what the problem is.
16:00 16-Jan-2012 New Gen for server2 is installed and tested.
14:00 13-Jan-2012 New generator arrived for Server2, Hope to install it ASAP.
16:25 08-Jan-2012 Server1 update, Server back online. All services are back up and running.
15:55 08-Jan-2012 Server1 update, New Strip installed & power is starting to be restored to servers, the server might take about 30min to come back online due to Raid rebuilding.
15:40 08-Jan-2012 Server1 update, Power Strip is being replaced, The reason the power does not fall back is there is too much currant being drawn on each strip, so when one goes down the other trips in overload
(eg, 2x 15A strips running at 10A each, 20A total that is 5A over.)

If this is not fixed soon we will be looking at moving the server over to another cabinet that is running load balanced.
14:50 08-Jan-2012 Server1 update, there is a power strip problem.
14:15 08-Jan-2012 Server1 Down, Looking into why.
08:30 02-Jan-2012 Server1 back online, some how the box had lost power, still looking into how a duel power supply box could loose power.
07:15 02-Jan-2012 Lost Conection the Server1, Looking into the problem.

All time stated above are in AEST (or AEDST) unless other wise stated. UTC Time Definition, CST Time Definition, AEST Time Definition