Posts Tagged cisco


Among the coffee mugs in my cupboard at home is one I’ve had for over 20 years.  It was a gift; if I remember right, a semi-joke gift in an office “Secret Santa”.

"Works and plays well with others"

“Works and plays well with others”. O RLY?

The slogan on it reads “Works and plays well with others”, and it’s a reference to one of the standard phrases seen on children’s school report cards.  It’s one of the standard mugs in my hot beverage rotation, and every time I use it I can’t help but think back to when it was new, and of how much has changed since those days.

It’s easy to treat a silly slogan on a coffee mug as little more than just a few words designed to evoke a wry grin from a slightly antisocial co-worker.  Sometimes it can take on a deeper meaning, if you let it.

For the last 6 months or more I’ve been working on transferring the function of our former demonstration facility in Brisbane to a location in Melbourne.  This has been fraught with problems and delays, not the least of which was an intermittent network fault into the network our systems are connected to.  Steady-state things would be fine; I could have an IRC client connected to a server in our subnet for days at a time.  When I actually try to do anything else (SSH, HTTP, etc), within about 5 minutes all traffic to the subnet would stop for a few minutes.  When traffic would pass again, it would stay up for five or so minutes then fail.  Wash, rinse, repeat.

It looked like the problem you get when Path MTU Discovery (PMTUD) doesn’t work and you have an MTU mismatch[1].  I realised that we had a 1000BaseT network that was connected to a 100BaseT switch port, so went around all my systems and changed where I was trying to use jumbo frames, but that made no difference to the network dropouts.  I found Cisco references to problems with ARP caches filling, but I couldn’t imagine that the network was so big that MAC address learning would be a problem (and if general MAC learning was constrained, why no-one else was having a problem).

Everything I could think of was drawing blanks.  I approached the folks who run the network we uplink through, and all they said was “our network is fine”.  I was putting up with the problem, thinking that it was just something I was doing and that in time we would change over to a different uplink and we wouldn’t have to worry any more.  My frustration at having to move everything out of the wonderful environment we had in Brisbane down to Melbourne, with its non-functional network, multiplied every time an SSH connection failed.  I actually started to rationalise that it was pointless to continue with setting up the facility in Melbourne; I’d never be able to re-create what I’d built in Brisbane, it would never be as accessible and useful, and besides no-one other than me had ever made good use of the z Systems gear in the Brisbane lab anyway.  Basically, I had lost confidence in myself and my ability to get the network fixed and the Melbourne lab set up.

Confidence is a mental strength, like our muscles which provide our physical strength.  Just like muscle, confidence grows from active use and wastes if underused.  Chemicals can boost it, and trauma can damage it.  Importantly though, confidence can be a huge barrier to a person’s ability to “work and play well with others” — too little confidence and one lacks conviction and decision-making; too much confidence and they appear overbearing and dictatorial.

Last week I was in Singapore for the z/VM, Linux on z, and KVM “T3” event.  Whenever I go to something like this I get fired up by all of the things that I’d like to work on and have running to demo.  The motivation to get new things going in the lab overcame my pessimism about the network connection (and lack of confidence), and I got in touch with the intern in charge of the network we connect through.  All I need, I said, is to look and see what the configuration of the port we connect into looks like.  We agreed to get together when I returned from Singapore, and try to work out the problem.

We got into the meeting, and I went over the problem in terms of how we experience it — a steady state that could last for days, then activity leading to three-minute lockouts.  I asked if I could see the configuration of the port we attached to… after a little bit of discussion about which switch and port we might be on, a few lines of Cisco CatOS configuration statements appeared in our chat session.  Straight away I saw:

switchport port-security

W. T. F.

Within a few minutes I had Googled what this meant.  Sure enough, it told the switch to monitor the active MAC addresses on that port and disable the port if “unknown” MACs appear.  There were no configured MACs, so it just remembered the first one it saw.  It explained why I could have a session running to one system (the IRC server) for ages, and as soon as I connected to something else everything stopped — the default violation mode is “shutdown”.  It explained why the traffic would stay down for three minutes and then begin again — elsewhere in the switch configuration was this:

errdisable recovery cause psecure-violation 180

If the switch disabled a port due to port-security violation, it would be automatically recovered after 180 seconds.

The guys didn’t really understand what this all meant, but it made sense to me.  Encouraged by my confidence that this was indeed the problem, they gave me the passwords to log on to the switch and do what I thought was needed to remove the setting.  A couple of “no” commands later and it was gone… and our network link has functioned perfectly ever since.

The real mystery for the other network guys was: why has this suddenly become a problem?  None of them had changed the network port definition, so as far as anyone knew the port was always configured with Port Security.  The answer to this question is, in fact, on our side.  To z/VM and Linux on z Systems people, networks come in two modes: “Layer 3” or “IP” mode, where the system only deals with IP addresses, and “Layer 2” or “Ethernet” mode, where the system works with MAC addresses.  In Layer 3 mode, all the separate IP addresses that exist within Linux and z/VM systems actually exist behind the MAC address of the mainframe OSA card.  In Layer 2 mode however, each individual Linux guest or z/VM stack gets its own MAC address.  When we first set up this network link and the z/VM and Linux systems there the default operating mode was Layer 3, so the network switch only saw one or two MAC addresses.  Nowadays though the default mode is Layer 2.  When I built new systems for moving everything down from Brisbane, I built them in Layer 2 mode.  Suddenly the network switch port was seeing dozens of different MAC addresses where it used to only see one or two, and Port Security was being triggered constantly.

This has been a learning experience for me.  Usually I don’t have any trouble pointing out where I think a problem exists and how it needs to be fixed.  Deep down I knew the issue was outside our immediate network and yet this time for some reason I lacked the ability, motivation, nerve, or whatever, to chase the folks responsible and get it fixed.  The prospect of trying to work with a group of guys who, based on their previous comments, really strongly thought that their gear was not the problem, was so daunting that it became easier to think of reasons not to bother.  Maybe it’s because I didn’t know for certain that it wasn’t something on our side — there is another problem in our network that definitely is in our gear — so I kept looking for a problem on our side that wasn’t there.

For the want of a 15-minute phone meeting, we had endured months of a flaky network connection.

On this occasion it took me too long to become sufficiently confident to turn my thoughts into actions.  Once I got into action though, it was the confidence I displayed to the network team that got the problem fixed.   For me, lesson learned: sometimes I need a prod, but I am one who “works and plays well with others”.


[1] I get that all the time when using the Linux OpenVPN client to connect to this lab, and got into the habit of changing the MTU manually.  Tunnelblick on the Mac doesn’t suffer the same problem, because it has a clever MTU monitoring feature that keeps things going.

Tags: , , , , , , , ,

Upgrading from Cisco

In case you weren’t aware, I am a VoIP nutcase.  I have an Asterisk phone system at home, and all the phones in the house are VoIP of some description (either real VoIP devices or analogue handsets through an ATA).  While I haven’t converted to VoIP as a replacement for PSTN, I have some connectivity to VoIP providers both here and overseas (and soon to be more, to help the phone-home situation while I’m overseas).

I’ve been a user of Cisco IP phones, buying 7960s and a couple of 7970s through a well-known internet site (maybe it starts with an “e”, not sure).  The phones have been excellent, and I’ve even written a few XML apps to supplement their use here.  The 7960s are getting a bit dated now, however, and I found myself contemplating buying 7971s (or even something newer, like the 7965 or 7975).  Before I committed myself further into the relationship with Cisco, though, I thought about what I was really getting out of using Cisco phones.

Like many users of second-hand Cisco gear, I only purchased the hardware.  I do occasionally succumb to a nagging feeling of being an “outlaw” (at least in the eyes of Cisco), but admittedly that feeling usually only comes when I find out that Cisco has released another new version of SIP software that I can’t get because I haven’t paid for SmartNet.  The last time I had this thought though, I had a realisation: even if I did pay for SmartNet, the only thing I’d get would be the firmware: Cisco will only support their phone software when connected to their CallManager server (yes, even the SIP firmware).  Anyone running Cisco phones against anything other than CUCM gets no support from Cisco in the event something doesn’t work–and based on the information floating around, the problems are many.

So basically I would be paying Cisco to allow me to run one of the worst SIP implementations in embedded existence, with no opportunity to report problems with it in my environment.  Hmm, let me think about that for a minute…

At around the same time, I happened across the NerdVittles site, and in particular the post where NerdUno nominated the Aastra 57i as the “World’s Best Asterisk Phone“.  I started to do some research into it, and was astounded at the level of support the manufacturer (a Canadian company which a few years ago acquired the telephony business of a little mob called Nortel) and the community provide for this phone and Asterisk.  Looking through the phone manual, I found functions that only work with Asterisk! I found a full set of integration scripts that provide XML applications, right through to automatic provisioning tools.  Possibly the best thing was that on the product page for their phones — right there on the page that descibes the product — are links to current versions of firmware, documentation, XML application development guides, even a Linux-based application to encrypt the phone configuration files.  Not hidden in some obscure hard-to-find portal, or behind a registration-only support site.

I started to think of the possibilities…  I’d be able to freely modify the phone configuration (even via a HTTP interface if I so chose), without having to make trial-and-error changes to a cryptic and totally undocumented configuration file.  I’d be able to write XML apps without having to do laborious debugging to cater for why the parser was choking on XML that was perfectly okay according to the documentation but apparently tripped over an undocumented field length restriction or character encoding limitation.  I could get access to things like Visual Voicemail, BLF, integration with Asterisk functions like day/night mode and call parking.  I could keep the phones up-to-date for new functions and bug fixes.  With a click of a mouse I could get proper Australian tones!

So, I decided to give one a try.  Finding nothing on that “e” site I went looking for a vendor locally, and found several places that would sell one to me (legitimate e-tailers, no less!  Zounds!  A VoIP phone with a warranty?  You jest!).  It took a while for my chosen vendor to source it for me, but I’ve had it now for a couple of weeks.  It’s probably going to take a while for it to live up to it’s full potential in my installation, but since that potential is so much greater than what I have been able to do with the Ciscos I think I’m already ahead.

More in the coming weeks as the Aastra settles in.

Tags: , , , , ,

BOM Radar for Cisco 7970

I’ve written an XML app to display rain radar images from the Australian Bureau of Meteorology on the screen of a graphics-capable Cisco IP phone. It seemed like it would be simple to do, and I couldn’t work out really why no-one had done it. Well it wasn’t hard to do, but I can see now why it hadn’t been done…

While the screen of the 7970 looks nice on it’s own, and is certainly a drastic improvement over the 7940/7960, it doesn’t have the resolution to be able to display anything useful other than text and simple images. The BOM images are 524×564, while the largest image the Cisco can show is 298×168.

Here’s an example of my app in action:


Because the phone’s image viewer can’t scroll a large image I’ve had to scale and crop it, losing a fair amount of detail in the process (I did try just scaling it to fit the entire image, but it was totally useless to view). Plus, one of the key attractions of the BOM radar site is the animated images, and the phone has no way to display animations.

So what do you think, lazyweb? I’ve learned a bit more about coding Cisco XML apps, but other than that have I wasted my time? I will put it up on my Projects site eventually (once I’ve put in some more error handling etc), but let me know with a comment here…

Tags: , ,

Cisco XML apps: things made of fail

Since I have a few Cisco phones around here, I’ve played with XML apps. I have written a timezone calculator, an LDAP phone directory lookup utility (which hooks into the “external directory” function of the phones), an app that uses Qantas’ WAP interface to get flight arrival/departure information, and the obligatory RSS reader. They work, in some cases very well, but the inconsistency of the XML interface between different levels of the Cisco firmware makes it a trying exercise.

My latest exercise was an update to the RSS reader I’ve used for ages. I found RSS2Cisco ages ago and have used it quite successfully, but I’ve never really been satisfied with its way of displaying the whole feed in one text page. It works well for news feeds, where all you get is a headline and a teaser, but for things like blogs it’s not suitable (you’re lucky to get through one posting before hitting the limit of a Cisco XML text page). I wanted an interface like a “normal” RSS reader, where it lists the items in the feed in a menu and you then choose an item to be displayed.

Sounded simple, and wasn’t too hard to hack rss2cisco around to make it do my bidding (it’s not optimal yet as every time you read an item it pulls down the entire feed again). The problem I faced was in making the thing work consistently between the 7960 phones and the 7970s.

All my phones are running fairly recent SIP code, but for some reason the 7960 has an ancient XML parser. By ancient, I mean that the level of the XML SDK it supports is tied back to Call Manager 3.0. The 7970s, on the other hand, have support for a much more recent SDK and support some of the fancy operations that you can’t do on a 7960 unless you’re running SCCP firmware. At first I thought that there might have been a hardware limitation and that Cisco couldn’t fit the extra smarts of a client for later SDKs, but the SCCP code can’t be that much simpler than SIP that they’d have more room to fit a better XML browser and all the other features the SCCP code has over SIP…

So the SIP firmware for 7960 has a junk XML browser. You’d think, then, that the 7970 was easier to work with than the 7960… Wrong! Valid XML that worked quite happily on the 7960 would fail with a cryptic “XML Error[4]: Parse Error” message. It took quite a bit of time and quite a bit of trial-and-error to work out some of the dependencies (32 seems to be a magic number, folks…).

Call Manager XML (CMXML) is supposed to be really simple, but I can only imagine how complex it might get to deliver an app with a consistent interface if you had a number of different phone models to support — I have only two, and I’m looking at two different versions of my app!

In their defence, Cisco have provided a way for the phone to identify itself and its SDK level when it makes a request. A set of HTTP headers identify the device, and one specifically states the SDK version supported by the phone client. Reading these headers would allow a developer to adjust the output of their app to cater for the various phones — one app, but multiple output capabilities.

It strikes me though as a heck of a lot of work for limited return. These are phones intended for corporate installations, so it’s almost a given that there will be a full-function computer at the same desk. Why would a company invest that much effort developing and supporting an internal application for a single platform that’s tied to desks, when they could write it as a web app and deliver it practically anywhere? I’m starting to see why the Internet is not exactly awash with sites selling CMXML apps…

Having said that though, I love my timezone calculator. With three button presses I can find out the time in any of my six favourite timezones, and I can find any timezone in the world with only a few more presses. An application somewhere on the web couldn’t be anywhere near that speedy for me, and a desktop app would have to be some kind of widget already running and configured (or be the KDE Clock applet, all it takes is a mouseover… shame I’m stuck with GNOME for my work desktop).

So I’m not too keen to apply much development effort to my XML apps. I will stick them on my development site some time soon, but I don’t think it’s worth the effort to keep them functional. The Qantas one, for instance, is totally dependent on the URL and query format of the Qantas WAP application, which is obviously subject to change at any time. I wonder sometimes if a WAP-XML gateway would be useful, but then I think about the effort of writing a system to translate pages delivered over a dying protocol to an interface that never got off the ground…

In case you’re curious what the RSS reader looks like:

and something a bit more voluminous from my blog:

Yes, I am a bit proud of it, even though it’s rubbish… 😉

Tags: , , ,

PoE again: this time, success!

I had pretty much forgotten about improving on the Power over Ethernet progress I mentioned previously. A couple of weeks ago I bought another 7970 that I successfully converted to SIP to run in the study, and I was considering buying a few 7961s or Linksys PoE-capable phones to use in other places. However, I got an e-mail from a reader whose success at using a hacked cable with his 7960G prompted me to have another go.

While I did a heap of research about PoE and IEEE 802.3af, the hints I got about using a hacked cable with a standard 802.3af switch to power a Cisco phone came almost exclusively from the wiki. Everything I’d seen about this trick relied on the use of a crossover cable to fix the problem where the phone using the Cisco pre-standard expects the power in the opposite polarity to that delivered by 802.3af.

When I’d had a go previously, the info I had told me that I had to get power onto the spare pairs in the Ethernet cable, because the Cisco pre-standard used the spare pairs for power and not the data pairs. This was a problem as my switch provided Type A PoE, which is power-over-data-pairs. In the end I figured that I’d have to come up with some kind of electronics to get the power off the data pairs and onto the spare pairs.

My friendly reader informed me, however, that Cisco pre-standard phones take power from the data pairs as well as the spare pairs! Nothing I’d seen indicated that this was a possibility. So I pulled out the hacked-up cable I’d used previously and gave it another try… but it didn’t work.

I tried a bunch of alternatives that I probably tried before as well. I tried putting the sense resistor across the spare pairs instead of the data pairs, I tried switching the spare pairs around. But, since others had only ever reported success with a crossover cable, it had never occurred to me to try a straight cable instead. A bit of resoldering later, another try, and it worked!

Tried it with all my 7960s and it worked fine. So it looks like some 802.3af switches put power on the pairs in the opposite polarity to others (which is not a problem usually, as 802.3af devices have a bridge rectifier that allows them to handle either polarity).

Thanks to my friendly reader, I now have a way to power all my Cisco phones via PoE! Yay! The only caveat (one that I’ve only seen briefly mentioned anywhere) is the extra load placed on the cable by the 25kohm sense resistor — doubt it’s significant, but over a few phones it might add up.

Tags: , ,