Well, a busy two days running FireBrick courses, and I have to say that Jon gave me a run for my money.
Probably the first time I have had a "me" on one of my courses! I must be a nightmare to train me!
He was always one step ahead of everyone else, and at least two slides ahead of the presentation, whilst always checking his emails.
That said, I like this - made the course a lot more fun, and a hell of a lot of suggestions and even bugs/typos found as a result, which is excellent.
It was a pleasure to do a course for someone that knew his stuff, and was there to learn.
Thanks Jon.
Thursday, 26 January 2012
Tuesday, 24 January 2012
No such thing as 7 hours fix
Our favourite telco offer a service that is a 7 hour fix for broadband. It costs extra money, as you would expect. We have not yet used that service.
Now, for some time we have said that any fault, even if on the normal 40 hour fix, or 20 hour enhanced care level, if they fail to meet that, should be treated as a 7 hour fix expensive fault. After all they are already in breach of the agreement and they have the resources (the engineers for the 7 hour fix) to fix things...
Tonight I had an interesting echat - a simple matter of a faulty line still not fixed. The engineer was booked for today but no-show. Engineer lied saying he contacted the end user to confirm it would not be tomorrow... Lying for any sort of gain being a criminal offence under the Fraud Act. They committed a crime, as I see it. They knowing lied for their own gain. The police should be involved.
Now, on an extensive echat with our favourite telco, Alok confirmed that there was absolutely no way an engineer could come out to fix this in the next 7 hours. No way at all...
What can I say - that means the service for 7 hour fix is a lie, a fraud, criminal.
I await their explanation on this matter.
And (as I said in the echat), good luck to Alok in his next job.
Just to clarify - they offer a 40 hour fix service. They offer a 7 hour fix service. But surely, 33 hours in to the 40 hour fix service it is then a 7 hour fix service? That is the very definition, is it not?
Now, for some time we have said that any fault, even if on the normal 40 hour fix, or 20 hour enhanced care level, if they fail to meet that, should be treated as a 7 hour fix expensive fault. After all they are already in breach of the agreement and they have the resources (the engineers for the 7 hour fix) to fix things...
Tonight I had an interesting echat - a simple matter of a faulty line still not fixed. The engineer was booked for today but no-show. Engineer lied saying he contacted the end user to confirm it would not be tomorrow... Lying for any sort of gain being a criminal offence under the Fraud Act. They committed a crime, as I see it. They knowing lied for their own gain. The police should be involved.
Now, on an extensive echat with our favourite telco, Alok confirmed that there was absolutely no way an engineer could come out to fix this in the next 7 hours. No way at all...
Adrian Kennard (19:07:17):
None,. so the 7 hour fix service you offer is a lie, yes?
Adrian Kennard (19:07:25):
You could not meet that under any circumstances?
Alokxxxxx (19:09:34):
I am afraid so, at this time we cannot deploy an engineer, Adrian.
None,. so the 7 hour fix service you offer is a lie, yes?
Adrian Kennard (19:07:25):
You could not meet that under any circumstances?
Alok
I am afraid so, at this time we cannot deploy an engineer, Adrian.
What can I say - that means the service for 7 hour fix is a lie, a fraud, criminal.
I await their explanation on this matter.
And (as I said in the echat), good luck to Alok in his next job.
Just to clarify - they offer a 40 hour fix service. They offer a 7 hour fix service. But surely, 33 hours in to the 40 hour fix service it is then a 7 hour fix service? That is the very definition, is it not?
Sunday, 22 January 2012
Favourite telco
Has anyone else looked at today's Dilbert and wondered if he is talking to his favourite telco?
Friday, 20 January 2012
Busy week
Finally - end of the week - get a Chinese and relax in front of TV maybe...
UKNOF was fun this week. Always nice to mingle with geeks, and was not a bad venue so thanks to BT for that. I love the chatting to ex demonites at the social - getting rare to find people that have been there and done that with every conceivable type of networking quirk you can imagine. Oh, and as per everyone that asked me at the social... Them:"We saw your blog"... Me:"Mark all packets as LCP"... Them:"Damn, was hoping it was something I could do on my CISCO"... What can I say? "Should have gone to FireBrick". Thanks BT for letting us get an edge over the competition...
We had a quirk worthy of that crowd this week... As you may know half of BT's 20CN network was broken when using native IPv6. We got the whole "IPv6 is not supported" story on that, lovely. It meant we had to pad PPP packets under 74 bytes to make it work. Mad. FireBricks FTW though, as ever. BT have fixed it (well done BT). But we still had it turned on for 20CN lines on our end.
So we get someone saying they cannot get email, but, and get this, only... (a) IPv6, (b) Technicolor router, and (c) windows. Even loading MacOS on the same hardware was fine, or changing the router to a Billion, worked, and IPv4 was fine!!!!
We turned off IPv6 sub 74 byte packet padding (as BT have fixed it) and it works. Looking at logs, both small and large packets fine, but some in the middle were "lost" somehow. My guess is padding 74 byte PPP created padded Ethernet which is fine under 64 bytes but when over must break something in windows. We have to do some careful testing to find the cause - may be technicolor and may be windows - who knows. I don't know what the rule (RFC) is on PPP to Ethernet if padding should be removed. I bet it is some checksum offloading somwhere. All I know is that (as per RFC) padding on PPP is always valid so not us :-) It is solved by this tweak, but I would love to know the cause. We have to test this in the lab...
The ex-demonites at UKNOF (sorry guys I am shite with names) had similarly bizarre stories from times past and that was fun.
Of course, I missed last train, but even a £145 taxi back home was cheaper than a premier inn in London and a train back next morning, and was my own bed even if it was past 1am.
And the good news - seems Tom's parents have him named on their insurance and he may be able to claim for his camera kit - well done and best of luck.
And then world IPv6 launch - yay! But that really is nothing to us - we are in the tenth year doing this shit, and we have been allocating /48's to all new customers for over a year with free IPv6 pre-configured router since before last year's World IPv6 Day event. My grand plan is a special for existing customers to get a cheap IPv6 router from us and move beyond our current 25% of lines with IPv6. Would be cool to push that to over 50% by June 6th... Lets see.
Have a good weekend all...
UKNOF was fun this week. Always nice to mingle with geeks, and was not a bad venue so thanks to BT for that. I love the chatting to ex demonites at the social - getting rare to find people that have been there and done that with every conceivable type of networking quirk you can imagine. Oh, and as per everyone that asked me at the social... Them:"We saw your blog"... Me:"Mark all packets as LCP"... Them:"Damn, was hoping it was something I could do on my CISCO"... What can I say? "Should have gone to FireBrick". Thanks BT for letting us get an edge over the competition...
We had a quirk worthy of that crowd this week... As you may know half of BT's 20CN network was broken when using native IPv6. We got the whole "IPv6 is not supported" story on that, lovely. It meant we had to pad PPP packets under 74 bytes to make it work. Mad. FireBricks FTW though, as ever. BT have fixed it (well done BT). But we still had it turned on for 20CN lines on our end.
So we get someone saying they cannot get email, but, and get this, only... (a) IPv6, (b) Technicolor router, and (c) windows. Even loading MacOS on the same hardware was fine, or changing the router to a Billion, worked, and IPv4 was fine!!!!
We turned off IPv6 sub 74 byte packet padding (as BT have fixed it) and it works. Looking at logs, both small and large packets fine, but some in the middle were "lost" somehow. My guess is padding 74 byte PPP created padded Ethernet which is fine under 64 bytes but when over must break something in windows. We have to do some careful testing to find the cause - may be technicolor and may be windows - who knows. I don't know what the rule (RFC) is on PPP to Ethernet if padding should be removed. I bet it is some checksum offloading somwhere. All I know is that (as per RFC) padding on PPP is always valid so not us :-) It is solved by this tweak, but I would love to know the cause. We have to test this in the lab...
The ex-demonites at UKNOF (sorry guys I am shite with names) had similarly bizarre stories from times past and that was fun.
Of course, I missed last train, but even a £145 taxi back home was cheaper than a premier inn in London and a train back next morning, and was my own bed even if it was past 1am.
And the good news - seems Tom's parents have him named on their insurance and he may be able to claim for his camera kit - well done and best of luck.
And then world IPv6 launch - yay! But that really is nothing to us - we are in the tenth year doing this shit, and we have been allocating /48's to all new customers for over a year with free IPv6 pre-configured router since before last year's World IPv6 Day event. My grand plan is a special for existing customers to get a cheap IPv6 router from us and move beyond our current 25% of lines with IPv6. Would be cool to push that to over 50% by June 6th... Lets see.
Have a good weekend all...
Friday, 13 January 2012
Bad luck to be superstitious
Well, it is, in that if you are not superstitious then the crap that happens is just crap that happens and not bad luck as such...
So, Friday 13th, and clearly the network switches in THN know they are about to be replaced next week as one of them starts going "iffy". Well that is the best explanation we have at present for flaky traffic between two boxes on the same switch this morning. As ever, a total failure would have everything falling back in seconds to the secondary systems, but "iffy" is the system engineers worst nightmare.
Of course it happens when we have a completely separate minor issue affecting incoming connection RADIUS on 20CN lines due to our favourite telco jumping the gun on a config change. Well, an issue that would be minor. Basically, it only affected new connections, a couple of lines, and would take us minutes to solve. It took longer because all hell broke loose when the switch went iffy, and as we dropped all sessions it meant nobody on 20CN could reconnect! Anyway, all sorted now.
The good news is the new rack is being kitted out - we have several gigabit fibres in already and a load more links to set up and move over, new switches, new routers and LNSs (FireBrick, of course). All moving to multiple gigabit operations hopefully by the end of the month.
Of course that is not the only "crap that happens", but that is a different story.
Happy Friday 13th.
So, Friday 13th, and clearly the network switches in THN know they are about to be replaced next week as one of them starts going "iffy". Well that is the best explanation we have at present for flaky traffic between two boxes on the same switch this morning. As ever, a total failure would have everything falling back in seconds to the secondary systems, but "iffy" is the system engineers worst nightmare.
Of course it happens when we have a completely separate minor issue affecting incoming connection RADIUS on 20CN lines due to our favourite telco jumping the gun on a config change. Well, an issue that would be minor. Basically, it only affected new connections, a couple of lines, and would take us minutes to solve. It took longer because all hell broke loose when the switch went iffy, and as we dropped all sessions it meant nobody on 20CN could reconnect! Anyway, all sorted now.
The good news is the new rack is being kitted out - we have several gigabit fibres in already and a load more links to set up and move over, new switches, new routers and LNSs (FireBrick, of course). All moving to multiple gigabit operations hopefully by the end of the month.
Of course that is not the only "crap that happens", but that is a different story.
Happy Friday 13th.
Thursday, 12 January 2012
80Mb/s FTTC
Well, it is impressive what you can do on a short bit of copper :-)
We have a number of customers on the FTTC 80Mb/s trial now. One has 79.7Mb/s and another has 57Mb/s. This is pretty impressive. The uplink is around 20Mb/s as well...
Seems the trial is not going badly - the lines are syncing up at nice high rates, and the rates are correctly coming through to us.
Seems a few minor teething troubles getting the profiles right in the middle somewhere - but this is a technical trial, so not at all surprising.
Fun...
We have a number of customers on the FTTC 80Mb/s trial now. One has 79.7Mb/s and another has 57Mb/s. This is pretty impressive. The uplink is around 20Mb/s as well...
Seems the trial is not going badly - the lines are syncing up at nice high rates, and the rates are correctly coming through to us.
Seems a few minor teething troubles getting the profiles right in the middle somewhere - but this is a technical trial, so not at all surprising.
Fun...
 |
| Actual speed test on 58M sync line |
Monday, 9 January 2012
RIPE PI
They have stopped talking to us now, which is a shame.
The problem is quite well summed up in the RIPE policy document on this. Basically, until recently, PI space was issued without any contract.
The problem is that, without a contract, there is no way to ensure the end user follows RIPE policies, as there is no way to ensure PI space is returned if the end user does not follow RIPE policies. This is indeed a problem.
The solution that the community has come up with is to, err, make a policy which requires end users to have a contract.
So the solution to not being able to enforce policies is make a new policy, and to pretend that this new policy is enforceable.
Once the new policy is followed (contract in place) then all policies are enforceable.
It is a paradox though. If the new policy is enforceable, then so are the rest, by whatever means the new policy is. That means a contract is not needed because RIPE do have a way to enforce RIPE policies, so the new policy is not needed! If RIPE don't have a way to enforce policies, then the policy to have a contract is not enforceable so pointless.
I am appalled at the way we as a community have tried to do this. I should have been more involved in the discussions on this as a RIPE member myself. I was not. I do agree that all new PI space should have a contract - that is fine.
The key thing here is the wording in the policy about the end users returning resources. If an end user does not follow this new policy but does not return the resources then they still have them. If RIPE could simply take back resources then there would be no need for a contract with existing PI holders, but the policy document clearly recognises that end users would have to actually return the resources.
If the end user still has them then RIPE NCC have an obligation to ensure the RIPE NCC database is correct for use by all its members. If the end user still has the PI space then RIPE NCC should correctly record that in the database - otherwise the database is useless.
So far we have a string of unanswered questions for RIPE NCC on this...
1. The contract is being forced by coercion or duress in that RIPE NCC are saying they will remove database records which will effectively break the Internet connectivity. Forcing someone in to a contract is against the principle of contracts and a contract formed under duress is not valid.
2. The contract does not offer the end user anything. It gives the PI space (which the end user already has) and applies restrictions and gives third party rights, but does not give the end user anything new. It does not guarantee a RIPE database record and explicitly does not guarantee Internet connectivity which is what the end user wants. A contract with no consideration is not a valid contract, which means that the third party rights to RIPE NCC are not valid either.
3. If the end user does not enter into this contract then they are in breach of RIPE policy. But there is no contract requiring them to adhere to RIPE policy so that does not matter. They do not have to return the PI space to RIPE. If the end user still has PI space then transit providers should honour it (even in the absence of RIPE database entry) and refusing to do it starts to raise issues that relate to the Communications Act and could involve OFCOM. Forcing people to accept IP announcements outside of the RIPE database is bad - the right thing to do is for RIPE NCC to ensure the database is correct and reflects that the end user has not returned the PI space.
Current plan is end user signs a contract, though has a cover note explaining that the contract is signed on the basis that the end user knows it is not valid or enforceable and that the end user has no intention of granting rights to RIPE NCC as a third party. If it can be shown the parties to the contract did not intend to grant rights to a third party then the contract cannot do so even if it is deemed otherwise to be valid.
I think, as a community, we should do better in future.
I think RIPE NCC should not be ignoring me now that we finally have a fairly good and well formed argument why what they are doing is wrong.
The problem is quite well summed up in the RIPE policy document on this. Basically, until recently, PI space was issued without any contract.
The problem is that, without a contract, there is no way to ensure the end user follows RIPE policies, as there is no way to ensure PI space is returned if the end user does not follow RIPE policies. This is indeed a problem.
The solution that the community has come up with is to, err, make a policy which requires end users to have a contract.
So the solution to not being able to enforce policies is make a new policy, and to pretend that this new policy is enforceable.
Once the new policy is followed (contract in place) then all policies are enforceable.
It is a paradox though. If the new policy is enforceable, then so are the rest, by whatever means the new policy is. That means a contract is not needed because RIPE do have a way to enforce RIPE policies, so the new policy is not needed! If RIPE don't have a way to enforce policies, then the policy to have a contract is not enforceable so pointless.
I am appalled at the way we as a community have tried to do this. I should have been more involved in the discussions on this as a RIPE member myself. I was not. I do agree that all new PI space should have a contract - that is fine.
The key thing here is the wording in the policy about the end users returning resources. If an end user does not follow this new policy but does not return the resources then they still have them. If RIPE could simply take back resources then there would be no need for a contract with existing PI holders, but the policy document clearly recognises that end users would have to actually return the resources.
If the end user still has them then RIPE NCC have an obligation to ensure the RIPE NCC database is correct for use by all its members. If the end user still has the PI space then RIPE NCC should correctly record that in the database - otherwise the database is useless.
So far we have a string of unanswered questions for RIPE NCC on this...
1. The contract is being forced by coercion or duress in that RIPE NCC are saying they will remove database records which will effectively break the Internet connectivity. Forcing someone in to a contract is against the principle of contracts and a contract formed under duress is not valid.
2. The contract does not offer the end user anything. It gives the PI space (which the end user already has) and applies restrictions and gives third party rights, but does not give the end user anything new. It does not guarantee a RIPE database record and explicitly does not guarantee Internet connectivity which is what the end user wants. A contract with no consideration is not a valid contract, which means that the third party rights to RIPE NCC are not valid either.
3. If the end user does not enter into this contract then they are in breach of RIPE policy. But there is no contract requiring them to adhere to RIPE policy so that does not matter. They do not have to return the PI space to RIPE. If the end user still has PI space then transit providers should honour it (even in the absence of RIPE database entry) and refusing to do it starts to raise issues that relate to the Communications Act and could involve OFCOM. Forcing people to accept IP announcements outside of the RIPE database is bad - the right thing to do is for RIPE NCC to ensure the database is correct and reflects that the end user has not returned the PI space.
Current plan is end user signs a contract, though has a cover note explaining that the contract is signed on the basis that the end user knows it is not valid or enforceable and that the end user has no intention of granting rights to RIPE NCC as a third party. If it can be shown the parties to the contract did not intend to grant rights to a third party then the contract cannot do so even if it is deemed otherwise to be valid.
I think, as a community, we should do better in future.
I think RIPE NCC should not be ignoring me now that we finally have a fairly good and well formed argument why what they are doing is wrong.
Subscribe to:
Posts (Atom)