Friday, April 18, 2008

April 18, 2008

DSL SERVICE RESTORED AT 5:45pm.


To All SpeedFactory Customers

In the early hours of this morning, our network monitoring system alerted us to a problem with our DSL circuits in Atlanta.

We reported the outage to AT&T, who acknowledged an internal error on their part. However, we were unable to make any significant progress until early morning. By the time it was resolved we had to work through five different parts of their organization to get it fixed.

I apologize for the loss of service, coming so quickly after last week’s outage. While this was entirely outside of our control, I know that your access to the Internet has been disrupted, and that is not acceptable. As your Internet Service Provider, my commitment to you is that we will work with AT&T to understand the cause of this outage, and ensure that it does not happen again.

Each and every customer is important to us, and I recognize that recent events will have shaken your confidence in our abilities to provide you with a reliable service, and maybe you’re questioning if we will continue offering services in the Atlanta region.

On behalf of myself and my team, I want to assure you that we are here to stay and we are working diligently to eliminate future problems. We are committed to winning back your confidence and providing you with a service that we feel proud of, and that meets your needs.

Sincerely,

Robert Ford
President
SpeedFactory, Inc.


CONTACT US AT: http://speedfactory.net/contact.php

FINAL UPDATE. CIRCUITS UP

We have confirmation from AT&T that all our circuits are up, which means that our service to DSL clients is fully recovered.

We thank you for your patience. Our CEO will send a note to customers briefly.

17:11 Update. Good signs

We are checking our network interfaces, and they show good signs of being up... this might be the beginning of total recovery...will keep you posted

16:17 Update. LAST UNTIL FURTHER FROM AT&T

It seems like AT&T cannot provide regular updates to us. We just got word that it may take them several more hours until this circuit is back. We are extremely sorry about that. We keep taking phone calls from customers and remain open, with our full technical capacity ready, but waiting from AT&T. There's no signal on our end, so they must work on their side to fix it first.

We have decided to post a new update only where there is real news. It does not make sense to bother you now with regular updates that have no relevant information.

We'll post as soon as there is something new to announce. Apologies again. We're doing everything in our hands to speed up this process... thank you for understanding us.

16:00 Update

No progress reported by AT&T. A backbone OC3 circuit to us is still down, and they keep telling us that they are working hard on it. Our account manager in AT&T is very resourceful, and has been on the line with us from the very first minute. However, it seems quite difficult to get them to involve the right people to get this resolved... we are hoping that we can provide better news every single time we write these updates...

15:30 Update

AT&T has recognized a minute ago that this failure is on their side, no other vendors. They are focusing their efforts now on restoring our OC3...

15:00 Update

We are opening tickets with other vendors as well in an attempt to ensure that any possible interdependencies are being addressed between them...

14:30 Update

Elevated to level 3 within AT&T... still same impact...

14:03 Update

One of the AT&T ports that serve SpeedFactory is up, but the rest are still down. We are passing this information up to AT&T account managers, at the same time that technical service works with us to fix the rest...

14:00 Update

We continue to work with AT&T on our DSL service issue. We have agreed with our AT&T account manager to open as many service tickets as DSL customers we serve, to create enough critical mass and get their service department attention. We have already talked to their management about it, and they seem committed to help us resolve it in the next few hours, but we have not received any definite diagnostic yet.

If we hear something new, we will post before 2:30pm EST.

13:20 Update

In order to give this issue its fullest attention I would like to extend my updates to every 1/2 hour.

At the bottom and top of the hour I will update this forum. I will update this thread immediately when the services have been restored.

Regards

Craig Statt

13:00 Update

We have just completed a call with AT&T and we are to understand that the issue is more complicated than what was first anticipated.

AT&T and Nextphase has now expanded our engineering team to tackle this problem. We are currently conducting a task force conference call as this is being typed.

Again we are working diligently to have this resolved.

Regards

Craig Statt

12:40 Update

Once again we are being assured that resolution is imminent. We continue to remain diligent and will update this forum as soon as we see results or have further information to share.

Regards

Craig Statt
Network Engineer
INS (a NextPhase Wireless Company)
856-227-4428

12:20 Update

AT&T representatives keeps reassuring us that they are on their way to fix this problem, but we have been monitoring our network and currently see no results. We are still here and pushing to get your services back. We will continue to update this thread.

Regards

Craig Statt
Network Engineer
INS (a NextPhase Wireless Company)
856-227-4428

12:00 Update

We have been advised by AT&T that we should be restored within "15 minutes" we will keep you posted and will continue to update you every 15 minutes.

Regards

Craig Statt
Network Engineer
INS (a NextPhase Wireless Company)
856-227-4428

11:40 Update

At the moment we are currently connected to 5 separate departments within AT&T in order to get the issue resolved. We are making every effort to push AT&T and we have escalated this issue to their management in order to restore your service ASAP.

Regards

Craig Statt
Network Engineer
INS (a NextPhase Wireless Company)
856-227-4428

11:20 Update

AT&T are still working internally to resolve the issue. We apologize for the inconvenience. We are doing everything in our power to restore our service ASAP.

Regards

Craig Statt
Network Engineer
INS (a NextPhase Wireless Company)
856-227-4428

11:00 Update

We are in permanent communication with AT&T, who are working hard to resolve this involuntary outage. We are committed to inform you every fifteen minutes with an update.

Regards

Craig Statt
Network Engineer
INS (a NextPhase Wireless Company)
856-227-4428

11:00 Update

We are in permanent communication with AT&T, who are working hard to resolve this involuntary outage. We are committed to inform you every fifteen minutes with an update.

Regards

Craig Statt
Network Engineer
INS (a NextPhase Wireless Company)
856-227-4428

10:30 Update

AT&T has admitted that the error is internal to AT&T and they are currently working to resolve it.

Hopefully this will be resolved shortly.

Regards

Craig Statt
Network Engineer
INS (a NextPhase Wireless Company)
856-227-4428

Speedfactory is aware and working on the current issues

Speedfactory is currently aware of the current connectivity issues. These issues are currently only affecting our DSL customers. Our technicians in Atlanta as well as New Jersey are currently working to resolve the issue. We hope to have a resolution shortly.

I will try to keep this thread updated with our status as we work through this issue.

Again we apologize for the inconvenience and please rest assured that Nextphase does have a strong strategy to resolve the recent issues in ATL.

Regards

Craig Statt
Network Engineer
INS (a NextPhase Wireless Company)
856-227-4428