Integration and Abstraction

While performing admirably at its intended job (many of the details of which still elude me as I’m no geomatics engineer) the software can be a little monolithic. It works as advertised, make no mistake, but its UI is a bit clunky and it doesn’t seem to integrate well into other applications without a great deal of massaging. [Please Note: I say this with respect for the fact that the software is easily the best available for GPS/GNSS infrastructure and it does seem as though Trimble is making strides towards modernizing their applications.]

One of my main goals with regard to the infrastructure software has been (while working within and around the known limitations of the software) to abstract account management, reporting, and our internal tech support processes from staff access to the actual servers themselves. There are many reasons why this is necessary in an operation this size:

Automation

It can be very difficult for staff to manage sales and account terms in ERP software alongside user accounts in the GPS Infrastructure. Automation is necessary to ensure that customers are getting the accounts and access they paid for and freeloaders are kept out.

Accountability

Having user access to the database abstracted through a custom management tool means a greater level of logging of historic changes to user access tables and moderation of the types of changes that can be made.

Stability

More staff accessing the servers means a greater possibility of staff accidentally closing the server applications (much of the software runs windowed) and making problematic changes. These sorts of things can interrupt end-user access. Abstraction means greater regulation of what software runs on the servers and the resources used by that software.

Security

Giving direct access to staff invites unintended use. Since simply opening a web browser to a questionable page in Windows can result in virus/malware infection it’s important to keep the list of users with access to the actual servers to a minimum.

This necessity resulted in the ongoing programming of an intranet web application (written primarily in Ruby on Rails) which I may discuss in more detail in another post.

Bolt-ons

One of the roadblocks in the abstraction process has been the lack of useful real-time output from the infrastructure software for monitoring purposes. Access logging occurs only after a user logs off. This has made it necessary for our technical support to call the server admins to obtain information on whether or not a station is streaming data or a user is online. This is not ideal and for reasons stated above, it’s even less ideal to give them server access simply to check the online users list.

Bolt-on #1: Google Maps

While not my idea, this was a good choice for (near) real-time stationary station connectivity status.  Using Google Maps allows not only our staff, but all users and the general public (required by Google’s terms of use) the ability to check our station connectivity status.

My implementation uses protocols available in the infrastructure software to obtain a list of stations and logs that to a static file on a periodic basis. This allows me to place the file off the servers (better for security) as well as to provide a very quick-loading source of data for the Google map (the server’s own protocols for obtaining this data are insanely slow).

I have a plan to implement more up-to-date station statistics to our tech support staff using a means other than Google Maps, but more on that another time.

Bolt-on #2: Who’s Online?

As mentioned earlier, getting a list of the users currently connected to our system requires access to the server. In particular, access to the User Interface software for Trimble’s NTRIP Caster service. The UI provides usernames of connected users, amounts of data transferred, mountpoints (data streams), source IP addresses and source ports. Unfortunately, owing to the fact that many of our accounts have serialized alphanumeric usernames (an important part of our ERP integration) the username itself doesn’t tell us much about who is connected. Ideally, we need an approach that allows us to cross-reference the usernames with our ERP to provide customer associations and to view the list of users via a management tool running on a separate machine.

I’ve heard of some administrators using screen scraping utilities to provide this data to their customers. This method would require the GUI to be maximized and foremost on the screen at all times. Additionally, the data would be an image (unsortable, unfilterable and unusable for anything but display).

My method requires a little more diligence but results in the type of data and access we need:

The most important part of this method is the tool TShark, part of the Wireshark package. TShark and Wireshark are network packet analysis tools. They (with the help of passive packet capturing tools) can apply filters to network traffic in realtime and output specific information about the data. For this purpose, I set up TShark to run realtime analysis on the same system running the NTRIP Caster, monitor the same ports, filter out HTTP Authentication data (NTRIP is based on HTTP and uses basic auth) on those ports and produce log files of selected fields (namely: source IP, source port, and username/password).

I then wrote a script on a separate server that obtains these authentication log files, uses SNMP to obtain a list of source IPs and ports for current connections to the NTRIP caster, compares the IPs and ports to those in the log files, and loads the most recent matching usernames into an array. It then does some database magic to match the usernames to customers and outputs the result as a JSON file which can be parsed using libraries available in most programming languages. This script is run by Cron every minute.

Using an AJAX intranet web page that reloads and parses the JSON file every minute, I can provide a list of current users to our technical support staff that’s internal to our company network and does not require direct access to the servers.

[Edit: It should be noted that TShark chews up RAM pretty quickly.  I have it restarting on a daily basis so it doesn't eat all the system memory.  There are probably other tools that can do the job, or you could write a custom parser that's more efficient if that's a concern.  I chose TShark because it was the easiest solution to implement out-of-the-box.]

One day, it was decided that one of the sales staff would be working long-term in a town where we had no office and he would need a small inkjet printer that could run in his car. He would be connecting to the internet using a mobile 3G ExpressCard and the printer would be connected to his laptop by Bluetooth. On his laptop, he would be running the corporate VPN software so that he could access email and use the ERP software to make sales and what not. However, his new special printer only supported PCL3 and nobody had yet determined how he would be printing invoices.

I was asked to look into it and started to examine the inner workings of the printing system in order to find a way. If I did not succeed, the salesman would be emailing himself invoices to print off, which would be an annoying extra step, to be sure.

First I tried making the ERP software print PCL3 directly, which was largely unsuccessful. I could get the template to print cleanly, but as the software was outputting the overlay in PCL5 or 6, I couldn’t get the text to print on the same page or even line up cleanly. There had to be a way to interpret and convert PCL data in much the same way that PostScript can be interpreted and converted to other formats.

After a great deal of searching, Google found me GhostPDL (new name of GhostPCL), which was exactly that–a PCL interpreter and converter. It is to PCL as Ghostscript is to PostScript. I tested this tool with the template files (stored in pcl) and it worked! I could convert a PCL5/6 template to PCL3 directly, or convert it to postscript and have Ghostscript render the PCL3 (seemed to look much nicer this way). I was still stumped, however, about how I would perform this operation on the fly without being able to modify the ERP software.

It was time to investigate the workings of the Common UNIX Printing System (CUPS). Typically, our ERP would print to a raw spool on CUPS which would then forward the raw PCL to a network-capable PCL printer. This new printer was bluetooth and spoke only to the salesman’s laptop. In addition, the salesman was not connected to the internet all the time. It would be hard for CUPS to find the printer each time it needed to spool to it.

First things first, to keep the laptop as secure as possible while allowing the printer to be shared, I installed the Print Services for UNIX component in Windows XP. This runs an LPD server on port 515 which is very easy for CUPS to connect to. That solved the communication issue with the bluetooth printer. However, the system was still only connected to the internet by a mobile 3G card and got a new IP address on almost every connection. I decided to solve this by using DHCP over the VPN connection to reserve the laptop a statically assigned IP address. Every time the VPN client software was running, the laptop would accept LPD connections on the same IP, so CUPS could always find it. This works very well.

Unfortunately, the data was still not being properly filtered to PCL3. This was the most difficult part of the ordeal. CUPS does not have a simple method of adding a filter on a particular print queue. It supports global filters on specific mime types and allows overrides in the print drivers, but no simple per-queue filter. So, I put together a PPD file with an appropriate *cupsFilter line, made some changes to the mime types so PCL was no longer captured under the application/vnd.cups-raw heading and voila (well, not quite that quick, but after a lot of tweaking and testing), things worked.

A number of other little things had to be changed, but they aren’t really worth mention independently.

Presumably the salesman will live happily ever after. At the very least, I’m entertained by this working as well as it has. When the overlay printed smoothly over a perfectly printed template the first time today, I was cheering aloud.

This has been a nerdy moment brought to you by the letters I and T and a lot of acronyms.

I’m still working at the same place as last year, which is nice. It’s the longest I’ve worked full time in a single IT position, and I’m very happy to be there. My boss is still amazing, and I’m keeping challenged. My main project as of the past couple/few months has been to integrate our GPS infrastructure backend with our ERP/accounting system and it continues to be fun. I’ve been using Ruby and Ruby on Rails lately and have been rediscovering a love for programming that was soured by PHP for some time. I’ve also been doing a bit of contract work on the side.

I miss school somewhat, but I don’t miss having a student budget. I’m still in debt, but slowly whittling it down while living reasonably well. It’s been a refreshing change to be able to buy what I want and give gifts without worrying about how I’m going to afford to feed myself. I’m pretty sure I’m not ready for that to end yet.

I’m in the white fedora.

Since the end of January, I’ve been learning to Lindy Hop with the U of C Swing Dance club and attending weekly dances with Swing Dance Calgary. Previously, I had never been very taken with or good at any particular styles of dance, but after just one class and one dance I was completely sold on Lindy. Having only learned some basic East Coast Swing in junior high school, I didn’t have any idea that there were different types of swing or that what we were learning at the time was really a regulated ballroom version of often far more entertaining classic swing styles.

The classes at the University consist of an hour or so of class followed by social dancing at The Den (which is where I can be found after 9PM every Tuesday). Since one can never get enough extra instruction, I signed on for a second beginners’ course with Swing Dance Calgary (I’ve had 2 lessons so far of that one) and next weekend I’m going to the Prairie Lindy EXchange (PLEX) in Edmonton which will be a weekend of dance events, activities and live music.

I met Heather’s friend Phil who seems pretty cool, and got to hang out with Jeremy on Sunday.

I contracted an evil type of Stye in my left eyelid on Sunday morning and am currently on antibiotics to combat it. I blame grungy Otaku touching things (doorknobs, desks, etc) with their unclean hands (things which I subsequently touched prior to rubbing my eye). There were some foul odours among certain of the crowd. I think Grant MacEwan’s health services or similar body should be contacted for next year’s event to hand out soap, antiseptic wipes, hand sanitizer, and pamphlets. The comic that appeared in the Otafest guide should be enlarged and posted at the registration desk.

I received an offer today for a 3 month Network Administration (and helpdesk, and jack of all trades tech support guy) contract which I accepted and will commence next Tuesday. Hooray! Props to tozetre for helping this to happen.

I gave a week’s notice to Grand and Toy and committed to at least a couple more Saturday shifts for Black’s, though I’m thinking it’d be pretty sweet to work the one day a week at Black’s for a couple more months and use the extra pay there to get a camera at employee discount or something. We’ll see.