When did 24/7 come to mean banking hours?

I live on a resort in the Santa Cruz Mountains, about seven miles from the nearest central office on overhead copper. Despite this, I had decent luck when I called up Speakeasy, who seemed reasonably geek-friendly, to arrange for a T1, some years ago.

Yes, there were occasional problems, and in my case, the vendors actually providing the service were Covad and Verizon (for that final [seven] mile[s]). Due to its dispatch hierarchy, there were several hour delays with Verizon, but that was nothing compared to what has happened beginning in the last year.

For one thing, reliability dropped through the floor, beginning around November, 2006. Since then, there have been serious problems with connectivity on average something like once a month. This was bad. It got worse.

This is a resort near the San Francisco Bay Area, about a half hour drive from San Jose. We see most of our business on weekends. Weekends, therefore, are when we most need the service. And it seems like failures are nearly always on Friday or the weekend. But Covad and Verizon have both instituted policies that mean–at least for me–that they only respond to “hard down” conditions on weekends. The vast majority of problems appear as intermittent, even when they are so severe as to render the service unusable.

Because I am a graduate student and a teacher, it is impossible for me to be home during business hours many days of the week. This has the effect that Covad is refusing to provide service on a problem that began last Friday night, and has gotten progressively worse, until this coming Friday.

Now perhaps I’m being silly. But I thought that 24/7 meant 24/7. Covad and Verizon seem to think it means banking hours, which have actually extended to include Saturdays since I was a child, beyond what these vendors are willing to provide now. It also strikes me that such restrictive policies are neither geek- nor business-friendly.

Am I being unreasonable?

2008-02-01 19:09:26 David called in reporting intermittency. I warm transfered him to Travis.

Thanks,
Dustin [contact information elided]
2008-02-01 19:19:01 I logged into the router, and it showed a bounce on the WAN side, I am opening a ticket with covad to get the smart jack looped, and perform intrusive testing if necessary. David is available 24/7 for ILEC dispatch, and intrusive testing can be performed between 5pm and 9am.

Thank you,
Travis [contact information elided]
2008-02-01 19:23:57 Backhaul installed
This circuit has had 16 previous trouble tickets
2008-02-01 19:23:57 Backhaul installed
Ticket created
2008-02-01 19:54:42 Backhaul installed
Status changed from NEW to OPEN-Monitor
Billable Dispatch changed from null to N
Ticket being worked. Pending update
2008-02-01 20:03:48 Backhaul installed
-Ticket opened for Intermittent Issues
-Port is currently UP and passing traffic both ways
-Errors seen in Line and Path CVs in current 24hr interval of DSLAM statistics. None seen incrementing at this time
-Checked DS1 card. In 15 min interval saw 27 CV-P, 3 ES-P and and 31 UAS-P
-Will open an ILEC task to pull PMs and test ISP.
—————————————————
Dslam End Point : LSGTCAXG-SL1.1.19.1
DSLAM Trunk Status: OK
Technology: DS1
Card Status: OK
Port Status: Up
Actual Port Speed: 1536 kbps
Get Status of all ports on the card

Current 24hr period Previous 24hr period
NearEnd Line CV: 5 0
NearEnd Path CV: 146 13
Bin Status: invalid valid
Time Elapsed(secs): 67206
NearEnd LOS: 6 0
NearEnd LOF: 0 0

Snapshot Data:
Received Bytes: 14 ( 2092181591 )
Upstream % utilization: 0.0010416667%
Transmitted Bytes: 685 ( 2942896194 )
Downstream % utilization: 0.050520834%
PDU Errors: 0 ( 6403146 )
Errored Seconds – Line: 0 ( 315 )
Errored Seconds – Path: 0 ( 4797 )
2008-02-01 20:19:11 Backhaul installed
ILEC task opened ISP. We will continue to follow up
2008-02-01 21:14:29 Tried to call David, but connection is down. Were are setting up an ILEC ticket to pull PM stats, and will have an update in 3 hours.

Thank you,
Travis [contact information elided]
2008-02-01 22:56:27 Backhaul installed
-Statused ILEC task
-There are no test points on the circuit to remotely test
-They have referred it to the CO to continue testing
-Will follow up soon
2008-02-02 00:28:25 Backhaul installed
Travis called for an update on this ticket

I called ILEC and talked to Sharron who said the ticket had been placed in no access ? she said she will send this back to the tester and call me back when she knows what is going on
2008-02-02 00:29:09 Called David after calling Covad, covad technician claimed no access on their own CO, so we are escalating with the ILEC. David asked that we call him before 9am only if we need to get the dispatch out.

Thank you,
Travis [contact information elided]
2008-02-02 01:40:11 Backhaul installed
-Contacted the ILEC
-Reported from pulling PM data they saw no errors on the circuit
-They could not loop to the CSU though
-Currently port is UP and passing traffic both ways
-No errors in current 24hr interval
-Setting ticket to CLEARED for 24hrs ISP. If issues persist please approve for a Covad FST to clear the site to check the CPE/IW for issues. Thank you
————————————————-

Dslam End Point : LSGTCAXG-SL1.1.19.1
DSLAM Trunk Status: OK
Technology: DS1
Card Status: OK
Port Status: Up
Actual Port Speed: 1536 kbps
Get Status of all ports on the card

Current 24hr period Previous 24hr period
NearEnd Line CV: 0 2634
NearEnd Path CV: 0 228
Bin Status: valid invalid
Time Elapsed(secs): 1355
NearEnd LOS: 0 9
NearEnd LOF: 0 1113

Snapshot Data:
Received Bytes: 1428 ( 2542355829 )
Upstream % utilization: 0.10625%
Transmitted Bytes: 3825 ( 3734138440 )
Downstream % utilization: 0.284375%
PDU Errors: 0 ( 6403471 )
Errored Seconds – Line: 0 ( 321 )
Errored Seconds – Path: 0 ( 4809 )
2008-02-02 08:38:14 Backhaul installed
Please schedule a Covad PBD to clear the I/W and hardware.

Avail – tonight after 6pm he is avail and all day tomorrow.
2008-02-02 08:40:09 David called in to gain an update on the Vendor testing. I let him know that they found a clean loop with no issues and noted that the next step is to schedule a PBD to clear the I/W and hardware.

Avail – tonight after 6pm he is avail and all day tomorrow.

I updated the Vendor ticket and we should have an update in 1 hour. David will be away from the premise this afternoon and will call in for an update.

Regards;
Jason [contact information elided]
2008-02-02 11:09:13 I called Covad ESG / Pamela to follow up on my last update to the CFI. She has scheduled a Covad PBD for Monday between 12pm-1pm.

I will follow up with David as to this dispatch.
Regards;

Jason [contact information elided]
2008-02-02 11:15:29 Backhaul installed
Scheduled Covad PBDispach for FST to check IW and CPE
Dispatch scheduled for earliest available on Monday, Feb 4th between 12-4pm.
Thank you.
2008-02-02 11:15:34 I called David at the number on file and in this ticket [phone number elided] and was not able to leave a message as the phone tree did not offer an option for it.

The current Covad FST that is set for Monday 2/4/2008 btw 12pm-4pm.

Jason [contact information elided]

2008-02-02 16:18:16 I will not be on site on Monday between 12pm-1pm, and a T1 is supposed to be available 24 hours per day, 7 days per week.

In a now rather long history of trouble with this circuit, it is rather striking how often problems arise on weekends. I am no longer feeling that contract obligations are being met when these vendors fail to offer service on weekends.

Please advise these vendors that I will treat a failure to respond in a timely manner as a breach of contract and I expect this problem to be resolved on Sunday at the latest. Thank you.
2008-02-03 02:59:34 Backhaul installed
Reviewing TT
There is a Dispatch set for Monday–easrliest they could find…
No New Updates…
2008-02-03 09:12:30 David called in that he needed a dispatch to happen today. I am calling Covad to try to arrange this. He can be reached at [phone number elided], which is VoIP. If I can’t get through and he doesn’t hear from me in about 25 minutes, he will call me back.

Thanks,
Blake [contact information elided]
2008-02-03 09:24:03 I called Covad and was informed that there were no dispatches available for intermittency today. They are putting a note that if anything opens up they will call us and David with that information. I asked that the scheduled dispatch for tomorrow be pulled in to the morning spot if possible. They have flagged that as well and will update us if that occurs. As it stands the dispatch is still locked for tomorrow 12-4pm. I called David to let him know. He will be trying to make arrangements for someone to be on site tomorrow.

Thanks,
[contact information elided]
2008-02-03 12:15:29 Needing to get some work done today, and seeing it as unlikely that Covad will come out today, I have left for my office at California State University, East Bay, in Hayward, where my telephone is [phone number elided]. I have left a note for Covad in the office here by the front gate. Their hours are 9 am – 5 pm.
2008-02-03 17:37:42 The dispatch was not pulled in and is still set for tomorrow 12-4pm. We will check first thing in the morning to see if this changes.

Thanks,
Blake [contact information elided]
2008-02-04 09:18:00 Backhaul installed
Department changed from Dispatch to TAC
Assignee changed from Dispatch to TeleXtend

Moving to correct queue.
Setting check again to follow up with dispatch.

Dslam End Point : LSGTCAXG-SL1.1.19.1
DSLAM Trunk Status: OK
Technology: DS1
Card Status: OK
Port Status: Up
Actual Port Speed: 1536 kbps

PM Stats:
Current 24hr period Previous 24hr period
NearEnd Line CV: 0 26
NearEnd Path CV: 0 153000
Bin Status: valid invalid
Time Elapsed(secs): 1901
NearEnd LOS: 0 59
NearEnd LOF: 0 0

Snapshot Data:
Received Bytes: 88127 ( 3157465460 )
Upstream % utilization: 0.40260416%
Transmitted Bytes: 645200 ( 2781485461 )
Downstream % utilization: 2.9473958%
PDU Errors: 0 ( 8461003 )
Errored Seconds – Line: 0 ( 346 )
Errored Seconds – Path: 0 ( 15664 )
Time Since Snapshot Counters Reset:
1 Min. 54 Sec.
2008-02-04 12:30:32 Backhaul installed
…….Previously:
1/2 –
-Ticket opened for Intermittent Issues
-Port is currently UP and passing traffic both ways
-Errors seen in Line and Path CVs in current 24hr interval of DSLAM statistics. None seen incrementing at this time
-Checked DS1 card. In 15 min interval saw 27 CV-P, 3 ES-P and and 31 UAS-P
– opened an ILEC task to pull PMs and test ISP.
2/2 -ILEC Reported from pulling PM data they saw no errors on the circuit.
Scheduled Covad tech to dispatch to the site to clear the prem.
Dispatch is scheduled for 2/4 between 12-4.
…….Currently:
Port is UP, 2 way traffic, no live errors or packet loss.
Waiting for an update from the tech / dispatch, will follow up in aprox 1 hour.

……….
Reply from 66.92.48.93: bytes=1472 time=75ms TTL=245
Reply from 66.92.48.93: bytes=1472 time=80ms TTL=245

Ping statistics for 66.92.48.93:
Packets: Sent = 358, Received = 358, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 73ms, Maximum = 288ms, Average = 79ms
……….
Client Circuit Number: 101-702-089
Client Name: David Benfell
Company Name: none
Partner Name: Speakeasy, Inc.
Service Brand: TeleXtend 1.5

;
DSLAM Trunk Status: OK
Technology: DS1
Card Status: OK
Port Status: Up
Actual Port Speed: 1536 kbps
;
Current 24hr period Previous 24hr period
NearEnd Line CV: 0 26
NearEnd Path CV: 1 153000
Bin Status: valid invalid
Time Elapsed(secs): 13402
NearEnd LOS: 0 59
NearEnd LOF: 0 0

;
Received Bytes: 156550 ( 3167883101 )
Upstream % utilization: 2.0380208%
Transmitted Bytes: 289478 ( 2908588614 )
Downstream % utilization: 3.76875%
PDU Errors: 0 ( 8461005 )
Errored Seconds – Line: 0 ( 346 )
Errored Seconds – Path: 0 ( 15665 )
;
Time Since Snapshot Counters Reset:
0 Min. 40 Sec.
2008-02-04 12:37:36 Backhaul installed
Part 2 of 2:
—-SCROLL UP———————————–
02/03/08 18:21:12 T1: Received alarm condition cleared
02/03/08 18:21:12 T1: Receiving yellow alarm (RAI)- remote loss of framing
02/03/08 18:20:44 T1: Line is unavailable, due to excessive errors
02/03/08 18:20:37 T1: Lost remote framing – sending yellow alarm (RAI)
02/03/08 18:20:35 >>WAN: T1 1 deactivated
02/03/08 18:20:35 Frame Relay-1 down: No Sync
02/03/08 18:20:35 T1: Receiving blue alarm (AIS) – all ones signal
02/03/08 18:20:22 T1: Received alarm condition cleared
02/03/08 18:20:07 FR: DLCI 16 statically defined
02/03/08 18:20:07 >>WAN: T1 1 activated at 1536 Kbps
02/03/08 18:20:07 T1: Remote framing recovered – clearing yellow alarm
02/03/08 18:20:05 T1: Received alarm condition cleared
02/03/08 18:20:05 T1: Receiving yellow alarm (RAI)- remote loss of framing
02/03/08 18:20:01 T1: Lost remote framing – sending yellow alarm (RAI)
———————————SCROLL DOWN———————————-
2008-02-04 12:37:36 Backhaul installed
Part 1 of 2:

T1 Line Statistics / Diagnostics

–Condition——————01:43—12:35—12:20—12:05—11:50—24 hours-
Errored Seconds 000 000 000 000 000 00001
Unavailable Seconds 000 000 000 000 000 00000
Severely Errored Seconds 000 000 000 000 000 00001
Bursty Errored Seconds 000 000 000 000 000 00000
Loss of Frame Count 000 000 000 000 000 00000
Bipolar Violation Count 000 000 000 000 000 00001

…………

WAN Event History
Current Date — 2/4/08 12:37:12 PM
-Date—–Time—–Event——————————————————
———————————-SCROLL UP———————————–
02/03/08 18:23:50 FR: DLCI 16 active
02/03/08 18:23:50 IP: DLCI 16 attached to 66.92.48.1
02/03/08 18:22:58 T1: Received alarm condition cleared
02/03/08 18:22:40 FR: DLCI 16 statically defined
02/03/08 18:22:40 >>WAN: T1 1 activated at 1536 Kbps
02/03/08 18:22:38 T1: Receiving yellow alarm (RAI)- remote loss of framing
02/03/08 18:22:36 –Device restarted—————————————–
02/03/08 18:22:31 FR: DLCI 16 active
02/03/08 18:22:31 IP: DLCI 16 attached to 66.92.48.1
02/03/08 18:21:28 T1: Received alarm condition cleared
02/03/08 18:21:21 FR: DLCI 16 statically defined
02/03/08 18:21:21 >>WAN: T1 1 activated at 1536 Kbps
02/03/08 18:21:21 T1: Line is available with acceptable quality
02/03/08 18:21:14 T1: Remote framing recovered – clearing yellow alarm
———————————SCROLL DOWN———————————-
——————————
2008-02-04 14:23:33 Backhaul installed
Part 1 of 2:
Tech. Name: Troy Truchon
Tech. ID#: 4530

Was an attempt made to contact EU: Yes, but received no answer @ [phone numbers elided]

Is this a pull in: no

Date: 2/4/2008
Departure time: 12:40
AM or PM job: pm
Arrival Time: 13:30
Finish Times: 14:10
Miles: 12

Was the Central Office cleared: no need.

Status of Service: NID sync Verified.

Contact Names (when applicable):
EU: David Benfell
ISP: Mick

Wiring Work Performed (when applicable): Troubleshooting at the NID.
Billable IW: yes Time: 25 Authorization: David Benfell
Stress of IW: n/a
Stress of ILEC Loop: n/a
ILEC Loop Length: n/a
NID Splitter installed: no

FST Suggestion for next step (when applicable): Reschedule if the problem persists..

Additional Notes about Installation: I arrived onsite, but the EU was not home. The EU?s neighbor said he wasn?t there, as his car wasn?t there. I connected a test CPE at the NID and contacted the ISP. The ISP gave me two other phone numbers but when I called them they both gave me a message saying they were disconnected. I had good sync and surf at the NID with no intermittency. I left a door hanger. This should be rescheduled after the EU can provide availability.

NOSS (snapshot/provisioning), Ping, Trace Route, Loop Tests (Bert, MLT, etc.) results posted:

DSLAM Statistics
Client Circuit Number: 101-702-089

Client Name: David Benfell
Company Name: none
Partner Name: Speakeasy, Inc.
Service Brand: TeleXtend 1.5

Data/Dsl
Port Provisioning:

Dslam End Point :

DSLAM Trunk Status: OK
Technology: DS1
Card Status: OK
Port Status: Up
Actual Port Speed: 1536 kbps
Received Bytes: 274466 ( 31720
2008-02-04 14:23:33 Backhaul installed
Part 2 of 2:
94813 )
Upstream % utilization: 1.0666667%
Transmitted Bytes: 1719642 ( 2961411433 )
Downstream % utilization: 6.683854%
PDU Errors: 0 ( 8462609 )
Errored Seconds – Line: 0 ( 349 )
Errored Seconds – Path: 0 ( 15676 )

Time Since Snapshot Counters Reset:
2 Min. 14 Sec.
C:Documents and SettingsCOVADTECH>ipconfig

Windows IP Configuration

Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . :
IP Address. . . . . . . . . . . . : 66.93.170.242
Subnet Mask . . . . . . . . . . . : 255.255.255.240
Default Gateway . . . . . . . . . : 66.93.170.241

Ethernet adapter Wireless Network Connection:

Media State . . . . . . . . . . . : Media disconnected

C:Documents and SettingsCOVADTECH>tracert google.com

Tracing route to google.com [72.14.207.99]
over a maximum of 30 hops:

1 <1 ms >WAN: T1 1 activated at 1536 Kbps
02/04/08 14:00:58 T1: Line is available with acceptable quality
02/04/08 14:00:51 T1: Remote framing recovered – clearing yellow alarm
02/04/08 14:00:49 T1: Receiving yellow alarm (RAI)- remote loss of framing
02/04/08 14:00:49 T1: Received alarm condition cleared
02/04/08 13:39:42 T1: Line is unavailable, due to excessive errors
02/04/08 13:39:35 T1: Lost remote framing – sending yellow alarm (RAI)
02/04/08 13:39:33 FR: DLCI 16 inactive
02/04/08 13:39:33 IP: DLCI 16 detached from 66.92.48.1
02/04/08 13:39:33 >>WAN: T1 1 deactivated
****************************

T1 Line Statistics / Diagnostics

–Condition——————06:19—14:35—14:20—14:05—13:50—24 hours-
Errored Seconds 000 000 000 632 628 01261
Unavailable Seconds 000 000 000 642 627 01269
Severely Errored Seconds 000 000 000 632 637 01270
Bursty Errored Seconds 000 000 000 001 000 00001
Loss of Frame Count 000 000 000 000 001 00001
Bipolar Violation Count 000 000 000 001 001 00003

Line Status: Normal Operation
Loopback Status:
2008-02-04 14:41:36 Backhaul installed
Part 2 of 2:
Clear – No Loopback Enabled

Tests… Normal
2008-02-04 14:44:13 Backhaul installed
Part 1 of 2:
Status changed from ASSIGNED to CLEARED
Ticket opened for Intermittent Issues
Errors seen in Line and Path CVs in current 24hr interval of DSLAM statistics. None seen incrementing at this time
Checked DS1 card. In 15 min interval saw 27 CV-P, 3 ES-P and and 31 UAS-P
ILEC ticket opened to pull PMs and test – NTF.
Scheduled Covad tech to dispatch to the site to clear the prem.
2/4 FST dispatched to prem – customer no show, phone numbers disconnected, FST able to surf at NID.

Current:
DSLAM shows CPE pingable, no errors.
CPE stats show bouncing – but no current errors.
Partner to verify customer connectivity – if no further problems found, ticket will auto close in 24 hours.
If still having issues, will need firm access for Covad dispatch to check IW and CPE.

**********************

Client Circuit Number: 101-702-089
Client Name: David Benfell
Company Name: none
Partner Name: Speakeasy, Inc.
Service Brand: TeleXtend 1.5 :
Dslam End Point : LSGTCAXG-SL1.1.19.1
DSLAM Trunk Status: OK
Technology: DS1
Card Status: OK
Port Status: Up
Actual Port Speed: 1536 kbps

Current 24hr period Previous 24hr period
NearEnd Line CV: 7 26
NearEnd Path CV: 93 153000
Bin Status: invalid invalid
Time Elapsed(secs): 21449
NearEnd LOS: 2 59
NearEnd LOF: 0 0
Received Bytes: 122753 ( 3175255927 )
Upstream % utilization: 1.20625%
Transmitted Bytes: 347668 ( 2994371600 )
Downstream % utilization: 3.4161458%
PDU Errors: 0 ( 8462749 )
Errored Seconds – Line: 0 ( 349 )
Errored Seconds – Path: 0 ( 15679 )

Time Since Snapshot Counters Reset:
0 Min. 53 Sec.

Ping statistics for 66.93.170.241:
Packets: Sent = 390, Received = 390, Lost = 0 (0% loss)
Approximate round trip times in mi
2008-02-04 14:44:13 Backhaul installed
Part 2 of 2:
lli-seconds:
Minimum = 73ms, Maximum = 134ms, Average = 78ms
2008-02-04 15:38:27 Why did the Covad technician not check in at the front office?
2008-02-04 16:48:40 David,
I called to speak with you but only reached the recording noting that there is no listing there and requested to go back to the directory. Due to security and insurance purposes, Technicians are instructed to go to the main premise to gain contact and trouble shoot. The technician did go out to and trouble shoot, but from your DMARC which is were the phone lines come in to our site. From that point, he noted that there was no intermittancy, packetloss or any physical loop issue.

If you would like for Covad to dispatch out again, we will have to set it up for when you or someone else is on site.

Regards;
Jason [contact information elided]
2008-02-04 23:38:38 I am keeping the issue of our continuing business relationship on a separate ticket. I think you’ll agree that the issue of this immediate technical problem should be kept separate.

I have arrived home and found Covad’s note, which indicates that he hooked up a test router and found no problems. As noted, this implies that the problem is either with the CPE or the wire in between the smart jack and the CPE.

My router/CPE was connected the entire time. I’m guessing [s]he was unable to detect that router (which, by the way, was, at least, working long enough for me to download the page to enter this update). As I noted in the note I left for Covad at the front office, where it was my intent that the technician would have been able to find someone to watch over him while in my space, another replacement CPE would now be the fourth.

If we think the problem is either the wire or the CPE, then I’m guessing it is the wire.
2008-02-05 00:18:46 The wire between the smart jack and the CPE is in two parts. One, in a grey sheath, runs from the smart jack to a “stupid” jack just inside a window. A regular ethernet cable connects the “stupid” jack to the CPE. I have tried replacing the latter cable and resetting the CPE. Once again, the connection came up. We’ll see if it holds….
2008-02-05 06:24:07 David,
I am sorry that it seems impossible or at least painful to get them out there. If this issue does prove to be the wiring, I am more than happy to setup a covad dispatch, but we just need to coordinate on a Friday, as per your last notes indicating your availiability, so that one can be sent out while you are on site. If the issue is another Router, then what that is telling me is that the power in your area is unstable and spikes. You would need to pick up a power conditioner to plug the router in so that the current spikes do not surge the hardware. I can go with either option, please let me know how you would like to proceed.

Regards;
Jason [contact information elided]
2008-02-05 10:19:51 Power is definitely unstable in the area. That is why I have had the router, along with all my systems, on UPS pretty much from the beginning.

The line stayed up less than ten minutes and had been, as of the time I left for school this morning, been down continuously since.

The present CPE is very new. Covad put it in the last time they were here, which I’m pretty sure was less than a month ago.

That leaves two possibilities that I see. One, obviously, is the wiring that Covad offers as a possibility. The other is that something plugged into the router is causing it behave erratically. Unfortunately, I have seen the latter possibility before, and my suspicions are heightened by the fact that the VoIP unit continues to function, providing phone service, over the very circuit that is purportedly down.

As to your other comments, I will respond in the other ticket.

Author: benfell

David Benfell holds a Ph.D. in Human Science from Saybrook University. He earned a M.A. in Speech Communication from CSU East Bay in 2009 and has studied at California Institute of Integral Studies. He is an anarchist, a vegetarian ecofeminist, a naturist, and a Taoist.

Leave a Reply