Integration and Abstraction

While performing admirably at its intended job (many of the details of which still elude me as I’m no geomatics engineer) the software can be a little monolithic. It works as advertised, make no mistake, but its UI is a bit clunky and it doesn’t seem to integrate well into other applications without a great deal of massaging. [Please Note: I say this with respect for the fact that the software is easily the best available for GPS/GNSS infrastructure and it does seem as though Trimble is making strides towards modernizing their applications.]

One of my main goals with regard to the infrastructure software has been (while working within and around the known limitations of the software) to abstract account management, reporting, and our internal tech support processes from staff access to the actual servers themselves. There are many reasons why this is necessary in an operation this size:

Automation

It can be very difficult for staff to manage sales and account terms in ERP software alongside user accounts in the GPS Infrastructure. Automation is necessary to ensure that customers are getting the accounts and access they paid for and freeloaders are kept out.

Accountability

Having user access to the database abstracted through a custom management tool means a greater level of logging of historic changes to user access tables and moderation of the types of changes that can be made.

Stability

More staff accessing the servers means a greater possibility of staff accidentally closing the server applications (much of the software runs windowed) and making problematic changes. These sorts of things can interrupt end-user access. Abstraction means greater regulation of what software runs on the servers and the resources used by that software.

Security

Giving direct access to staff invites unintended use. Since simply opening a web browser to a questionable page in Windows can result in virus/malware infection it’s important to keep the list of users with access to the actual servers to a minimum.

This necessity resulted in the ongoing programming of an intranet web application (written primarily in Ruby on Rails) which I may discuss in more detail in another post.

Bolt-ons

One of the roadblocks in the abstraction process has been the lack of useful real-time output from the infrastructure software for monitoring purposes. Access logging occurs only after a user logs off. This has made it necessary for our technical support to call the server admins to obtain information on whether or not a station is streaming data or a user is online. This is not ideal and for reasons stated above, it’s even less ideal to give them server access simply to check the online users list.

Bolt-on #1: Google Maps

While not my idea, this was a good choice for (near) real-time stationary station connectivity status.  Using Google Maps allows not only our staff, but all users and the general public (required by Google’s terms of use) the ability to check our station connectivity status.

My implementation uses protocols available in the infrastructure software to obtain a list of stations and logs that to a static file on a periodic basis. This allows me to place the file off the servers (better for security) as well as to provide a very quick-loading source of data for the Google map (the server’s own protocols for obtaining this data are insanely slow).

I have a plan to implement more up-to-date station statistics to our tech support staff using a means other than Google Maps, but more on that another time.

Bolt-on #2: Who’s Online?

As mentioned earlier, getting a list of the users currently connected to our system requires access to the server. In particular, access to the User Interface software for Trimble’s NTRIP Caster service. The UI provides usernames of connected users, amounts of data transferred, mountpoints (data streams), source IP addresses and source ports. Unfortunately, owing to the fact that many of our accounts have serialized alphanumeric usernames (an important part of our ERP integration) the username itself doesn’t tell us much about who is connected. Ideally, we need an approach that allows us to cross-reference the usernames with our ERP to provide customer associations and to view the list of users via a management tool running on a separate machine.

I’ve heard of some administrators using screen scraping utilities to provide this data to their customers. This method would require the GUI to be maximized and foremost on the screen at all times. Additionally, the data would be an image (unsortable, unfilterable and unusable for anything but display).

My method requires a little more diligence but results in the type of data and access we need:

The most important part of this method is the tool TShark, part of the Wireshark package. TShark and Wireshark are network packet analysis tools. They (with the help of passive packet capturing tools) can apply filters to network traffic in realtime and output specific information about the data. For this purpose, I set up TShark to run realtime analysis on the same system running the NTRIP Caster, monitor the same ports, filter out HTTP Authentication data (NTRIP is based on HTTP and uses basic auth) on those ports and produce log files of selected fields (namely: source IP, source port, and username/password).

I then wrote a script on a separate server that obtains these authentication log files, uses SNMP to obtain a list of source IPs and ports for current connections to the NTRIP caster, compares the IPs and ports to those in the log files, and loads the most recent matching usernames into an array. It then does some database magic to match the usernames to customers and outputs the result as a JSON file which can be parsed using libraries available in most programming languages. This script is run by Cron every minute.

Using an AJAX intranet web page that reloads and parses the JSON file every minute, I can provide a list of current users to our technical support staff that’s internal to our company network and does not require direct access to the servers.

[Edit: It should be noted that TShark chews up RAM pretty quickly.  I have it restarting on a daily basis so it doesn't eat all the system memory.  There are probably other tools that can do the job, or you could write a custom parser that's more efficient if that's a concern.  I chose TShark because it was the easiest solution to implement out-of-the-box.]