Home, 2024 2023 2022 2021 2020 2019 2018 2017 2016 2015 2014 2013 2012 2011 2010 2008 2007 2006 2005
Haddow Internet Network Status History For 2009
|All is back online since 02:00, telstra has reported there was a L2 link problem from exchange to LNS server.
|Aus server link is going up and down, trying to trace down the problem.
|Aus servers back on mains power, we are going to have to replace the Batt in the UPS, They are over 2 years old.
|Aust servers back up and running, we are running of Gen Power, trying to trace down why the UPS gave out.
|Aust Server UPS has reported a power problem and is shuting down the servers, No Staff at the office, will get all back on line ASAP.
|Due to a big storm in the area we have lost mains power to the Aust Servers, Running on Back for now.
|All is back up and running as it should, still trying to find what the probem is.
|We are seeing the same problem again, we are trying to trace down what the problem is, it looks like the problem is between the Dslam at the Aust Exchange and the Auth server.
|Since 14:30 we have seen a lot of large ping times to the Australia Server, but all is back to the way it should be for now.
|There will be network maintenance in the USA Data centre again on the 22-Dec-2009 between 0600UTC - 1200UTC, it will be for upgrading the code and hardware on the routers within the network, This work will require rolling down times of the routers in the centre (Updates to posted here).
|Update: all back online and working (THEY HOPE).
|Update from the main provider: 23:35UTC : A data center technician preparing a patch cable required for the maintenance somehow managed to knock loose a network cable. Causing a network issue for some customers. We have re-requested that the core stack gear not be touched until the appropriate time.
|Shome how the USA server stoped responding for 10 min from 10:10am today, we are looking into the it.
|There will be network maintenance on one of the USA Data centers on 07-Dec-2009, this will occur between 0000UTC and 0200UTC, we have been told that any down time should be about max 10 min.
|Update, USA Auth. has blocked that server to all access to any Aust servers.
|The provider for the main link can not stop the problem, For some unknows resion, so we are looking at changing providers ASAP.
|Update, we are trying to get the ip blocked before it comes into Aust at all, the company that owns the network clame they are working on it, That was friday, it is now monday and they are still hamering away at the link. This type of atack is know as a DoS.
|Update, we have sent a email to the abuse auth and hope they fix it asap.
|We are seeing problems on the Aust server, we traced it down to a atack from a USA computer, it was only trying to make free phone calls via the voip server, we have derouted the ip address at the router but are still seeing traphic.
|Some people may have see that there site was down for about 1min, this was due to a lot of log file changes needed to be done on one of the servers, all is fine and working now.
|Daylight savings Starts in Australia today, all computer clocks changed with no problems.
|Some of the problems a fix, but there are still some problems at the DC.
|There are some routing problems in one of Sydney's DC.
|Power restored to Australia Servers, all back on line but may be slow while HDD checks are going on.
|UPS is reporting low battery, shuting down servers for now.
|Loss of mains power to Australia Server Running on UPS for now.
|Update, All went with out a problem.
|There will be an outage on the 2nd of Aug 2009 at the Data Center that host server1, this is the final step of the core switch upgrade, this outage will happen between 2100UTC and 2300UTC, there will be about 4 outages lasting up to 10 Min each.
|Australia Server Main Link back online, dslam had rebooted AGAIN.
|Australia Server Main Link down, Looking into the problem.
|The Core router upgrade has been done with not much problems, there will be another outage on the 2nd Aug to complete it..
|We have had notice from the USA Data Center, that there will be a Network Upgrade, on "Sunday July 26 and will take place between 2100UTC and 0100UTC". We anticipate about 3 to 4 network outages with a time out of about max 10 Min each. This outage is to move from 100Mb to 1Gb links, so we will have additional redundancy.
|We had reports that people could not access there E-mails and Websites, there has been a multi-gigabit DDoS atteck to some routers in the USA and Australia. This means that The USA server was still online but some people could not access it.
|Haddow Internet Server back online, all back up and running, we are checking for any problems.
|Update "We've completed the upgrade, and VPSs are starting back up now. The raid array is rebuilding on the server, which is going to cause some load issues, performance will be degraded for a while until that completes."
|Update "Upgrade compleated, Rebooted & bringing servers back on line"
|Update "There is some memory errors, and this host has had some heavy i/o lately. We are upgrading Xen, allocating more memory, and will have the server back online shortly"
|Update "Master server has been rebooted still waighting for the H.I. Server to come back online"
|Report from USA hosting that that the master server needs to be rebooted
|All back online, some log files had to be forced to roll over, but all seams to be back online and running, still weighting on a report onto why it happen in the first place
|Server responding, checking for any problems.
|Host Server back online, still no response to access on the server
|USA has fully stop responding, Still weighting for a update.
|USA is still pinging but have no access to it yet, weighting for a update.
|Problem reported by hosting company about master server in usa, they clam they have to reboot it.
|Problem with USA server, it is still pinging but nothing else is working, trying to report it to DC.
|There is a problem with one of the Aus servers, to fix the problem quickly, we rebooted the server, this means that some dns request were a bit slow but all back online. (Downtime 2 min).
|All server clocks set back the right time, some web sites needed changes due to hard coding, but all are back to AEST.
|Australia Daylight savings ends Sunday morning, set you clocks back 1 hr before you go to be Sat.
|All back online, the report is "there was a bit of a IO Load on the host server, removed the cause".
|Support have been called and we are hoping all will be online ASAP, error message "Xen failed to start. Error: Device 0 (vif) could not be connected. Hotplug scripts not working".
|After a "Reboot" and a "Power cycle" the server did not come back on line, we are looking into the problem.
|Problem on Main server, it will requier a reboot.
|All services in Dallas back on line, this is the first major problem we have had, and hope it does not happen again,
Services affected Main Web, Mail, and Pop.
|Upstream provider has reported "Update: looks like there may be a spanning tree loop after a new switch was added. This loop caused some switch interfaces to be shutdown. We disconnected the switch causing the problem. And now we are rebooting switches if those interfaces have been shutdown to re-enable them."
|Provider has reported "There is some kind of networking issue at the Dallas data center", we hope to have all serviecs back on line ASAP.
|All services in Dallas are dowm, we are looking into the problem.
|Australia Main server back online, Due to problems we had to replace the mother board. the server has gone from a 1.6Ghz to a 1.3Ghz board, BUT we will still have to change the system to another box down the track (as the usb is only v1.1 not v2) this is only for the backup of the hdd's.
The only services affected are backup mail, Support pages, backup VOIP, haddow.com, and backup DNS services. All other services were still running fine from the main server in USA.
|Australia Main server down, we are trying to get it back online ASAP, it looks like the server went crazy at about 0100 and then rebooted its self at 0730, but due to problems it did not come back up.
|Australia Main Link back on line, looks like TELSTRA DSLAM problem AGAIN.
|Australia Main Link down, Looking into the problem.
|Aust server is being replace, over the next few days the server may have a few down times, we are hoping all goes well with no problems.