Home, 2024 2023 2022 2021 2020 2019 2018 2017 2016 2015 2014 2013 2012 2011 2010 2008 2007 2006 2005

Haddow Internet Network Status History For 2009
08:00 19-Dec-2009 All is back online since 02:00, telstra has reported there was a L2 link problem from exchange to LNS server.
21:00 18-Dec-2009 Aus server link is going up and down, trying to trace down the problem.
18:30 17-Dec-2009 Aus servers back on mains power, we are going to have to replace the Batt in the UPS, They are over 2 years old.
18:00 17-Dec-2009 Aust servers back up and running, we are running of Gen Power, trying to trace down why the UPS gave out.
17:45 17-Dec-2009 Aust Server UPS has reported a power problem and is shuting down the servers, No Staff at the office, will get all back on line ASAP.
17:43 17-Dec-2009 Due to a big storm in the area we have lost mains power to the Aust Servers, Running on Back for now.
21:20 16-Dec-2009 All is back up and running as it should, still trying to find what the probem is.
21:00 16-Dec-2009 We are seeing the same problem again, we are trying to trace down what the problem is, it looks like the problem is between the Dslam at the Aust Exchange and the Auth server.
17:00 16-Dec-2009 Since 14:30 we have seen a lot of large ping times to the Australia Server, but all is back to the way it should be for now.
12:00 10-Dec-2009 There will be network maintenance in the USA Data centre again on the 22-Dec-2009 between 0600UTC - 1200UTC, it will be for upgrading the code and hardware on the routers within the network, This work will require rolling down times of the routers in the centre (Updates to posted here).
12:00 07-Dec-2009 Update: all back online and working (THEY HOPE).
11:00 07-Dec-2009 Update from the main provider: 23:35UTC : A data center technician preparing a patch cable required for the maintenance somehow managed to knock loose a network cable. Causing a network issue for some customers. We have re-requested that the core stack gear not be touched until the appropriate time.
10:30 07-Dec-2009 Shome how the USA server stoped responding for 10 min from 10:10am today, we are looking into the it.
14:00 03-Dec-2009 There will be network maintenance on one of the USA Data centers on 07-Dec-2009, this will occur between 0000UTC and 0200UTC, we have been told that any down time should be about max 10 min.
10:00 23-Nov-2009 Update, USA Auth. has blocked that server to all access to any Aust servers.
18:00 16-Nov-2009 The provider for the main link can not stop the problem, For some unknows resion, so we are looking at changing providers ASAP.
10:30 16-Nov-2009 Update, we are trying to get the ip blocked before it comes into Aust at all, the company that owns the network clame they are working on it, That was friday, it is now monday and they are still hamering away at the link. This type of atack is know as a DoS.
22:00 13-Nov-2009 Update, we have sent a email to the abuse auth and hope they fix it asap.
21:20 13-Nov-2009 We are seeing problems on the Aust server, we traced it down to a atack from a USA computer, it was only trying to make free phone calls via the voip server, we have derouted the ip address at the router but are still seeing traphic.
10:00 29-Oct-2009 Some people may have see that there site was down for about 1min, this was due to a lot of log file changes needed to be done on one of the servers, all is fine and working now.
03:01 04-Oct-2009 Daylight savings Starts in Australia today, all computer clocks changed with no problems.
18:23 04-Sep-2009 Some of the problems a fix, but there are still some problems at the DC.
17:15 04-Sep-2009 There are some routing problems in one of Sydney's DC.
21:54 25-Aug-2009 Power restored to Australia Servers, all back on line but may be slow while HDD checks are going on.
21:10 25-Aug-2009 UPS is reporting low battery, shuting down servers for now.
20:18 25-Aug-2009 Loss of mains power to Australia Server Running on UPS for now.
10:05 03-Aug-2009 Update, All went with out a problem.
Future Outage
03-Aug-2009		 There will be an outage on the 2nd of Aug 2009 at the Data Center that host server1, this is the final step of the core switch upgrade, this outage will happen between 2100UTC and 2300UTC, there will be about 4 outages lasting up to 10 Min each.
15:05 30-Jul-2009 Australia Server Main Link back online, dslam had rebooted AGAIN.
14:55 30-Jul-2009 Australia Server Main Link down, Looking into the problem.
12:00 27-Jul-2009 The Core router upgrade has been done with not much problems, there will be another outage on the 2nd Aug to complete it..
21:01 23-Jul-2009 We have had notice from the USA Data Center, that there will be a Network Upgrade, on "Sunday July 26 and will take place between 2100UTC and 0100UTC". We anticipate about 3 to 4 network outages with a time out of about max 10 Min each. This outage is to move from 100Mb to 1Gb links, so we will have additional redundancy.
14:05 16-Jul-2009 We had reports that people could not access there E-mails and Websites, there has been a multi-gigabit DDoS atteck to some routers in the USA and Australia. This means that The USA server was still online but some people could not access it.
13:00 14-Jul-2009 Haddow Internet Server back online, all back up and running, we are checking for any problems.
12:36 14-Jul-2009 Update "We've completed the upgrade, and VPSs are starting back up now. The raid array is rebuilding on the server, which is going to cause some load issues, performance will be degraded for a while until that completes."
12:05 14-Jul-2009 Update "Upgrade compleated, Rebooted & bringing servers back on line"
11:36 14-Jul-2009 Update "There is some memory errors, and this host has had some heavy i/o lately. We are upgrading Xen, allocating more memory, and will have the server back online shortly"
10:55 11-Jul-2009 Update "Master server has been rebooted still waighting for the H.I. Server to come back online"
10:45 14-Jul-2009 Report from USA hosting that that the master server needs to be rebooted
02:00 11-Jan-2009 All back online, some log files had to be forced to roll over, but all seams to be back online and running, still weighting on a report onto why it happen in the first place
01:34 11-Jun-2009 Server responding, checking for any problems.
01:15 11-Jun-2009 Host Server back online, still no response to access on the server
00:57 11-Jun-2009 USA has fully stop responding, Still weighting for a update.
00:55 11-Jun-2009 USA is still pinging but have no access to it yet, weighting for a update.
00:45 11-Jun-2009 Problem reported by hosting company about master server in usa, they clam they have to reboot it.
23:50 10-Jun-2009 Problem with USA server, it is still pinging but nothing else is working, trying to report it to DC.
15:30 26-May-2009 There is a problem with one of the Aus servers, to fix the problem quickly, we rebooted the server, this means that some dns request were a bit slow but all back online. (Downtime 2 min).
07:00 05-Apr-2009 All server clocks set back the right time, some web sites needed changes due to hard coding, but all are back to AEST.
15:00 03-Apr-2009 Australia Daylight savings ends Sunday morning, set you clocks back 1 hr before you go to be Sat.
09:44 31-Mar-2009 All back online, the report is "there was a bit of a IO Load on the host server, removed the cause".
09:15 31-Mar-2009 Support have been called and we are hoping all will be online ASAP, error message "Xen failed to start. Error: Device 0 (vif) could not be connected. Hotplug scripts not working".
09:10 31-Mar-2009 After a "Reboot" and a "Power cycle" the server did not come back on line, we are looking into the problem.
09:00 31-Mar-2009 Problem on Main server, it will requier a reboot.
16:22 11-Feb-2009 All services in Dallas back on line, this is the first major problem we have had, and hope it does not happen again,
Services affected Main Web, Mail, and Pop.
16:00 11-Feb-2009 Upstream provider has reported "Update: looks like there may be a spanning tree loop after a new switch was added. This loop caused some switch interfaces to be shutdown. We disconnected the switch causing the problem. And now we are rebooting switches if those interfaces have been shutdown to re-enable them."
15:45 11-Feb-2009 Provider has reported "There is some kind of networking issue at the Dallas data center", we hope to have all serviecs back on line ASAP.
15:30 11-Feb-2009 All services in Dallas are dowm, we are looking into the problem.
12:00 10-Feb-2009 Australia Main server back online, Due to problems we had to replace the mother board. the server has gone from a 1.6Ghz to a 1.3Ghz board, BUT we will still have to change the system to another box down the track (as the usb is only v1.1 not v2) this is only for the backup of the hdd's.
The only services affected are backup mail, Support pages, backup VOIP, haddow.com, and backup DNS services. All other services were still running fine from the main server in USA.
09:00 10-Feb-2009 Australia Main server down, we are trying to get it back online ASAP, it looks like the server went crazy at about 0100 and then rebooted its self at 0730, but due to problems it did not come back up.
11:48 25-Jan-2009 Australia Main Link back on line, looks like TELSTRA DSLAM problem AGAIN.
11:39 25-Jan-2009 Australia Main Link down, Looking into the problem.
16:00 18-Jan-2009 Aust server is being replace, over the next few days the server may have a few down times, we are hoping all goes well with no problems.

All time stated above are in AEST (or AEDST) unless other wise stated. UTC Time Definition, CST Time Definition, AEST Time Definition