REST API – Locking it Down
A look at using Secure Communications and authentication restrictions to trigger Geneos commands programmatically over the Commands REST API.
I’ve had occasion to work with the Gateway Command Rest API often. A client had inquired about the security of it, and while I knew there are authentication options, I wanted to kick the tires a bit myself. Further, the secure communications functionality available in Geneos seemed worth looking at. Often, I am working in test scenarios, POCs, and building integrations, seldom do I implement security and authentication in my work with Geneos. I wanted a chance to play with these two pieces of functionality and to see how it all worked together, bringing my foray into working with the Gateway Command REST API to a close.
This writing is informative as a standalone read, although it does build on the basics outlined in other publications. Users may get more out of it and be better able to understand the content after having worked with the REST API a bit and read the other two blogs on the REST API – “REST API – Not Just For Snoozing!”, and “REST API – What am I doing wrong?”. Those blogs speak to different topics and focus more on the basics of the REST API.
A big help in all this has been the API and HTTP request tool – Postman. It lets you create one-off HTTP requests for testing purposes, see the responses you get back, save your built requests, and much more, all in a nice easy to use GUI. I’ll be using it throughout this publication.
There really are two different pieces of functionality around security, gateway authentication and secure connections. Authentication focuses on entitlements and access restriction/validation, whereas secure communications is about encrypting the communications sent to the gateway in any form, including over the REST API.
Without further ado…
Authentication - Gateway authentication means authenticating the requests asking to run commands against the gateways authentication policy. This ties in to the entitlements policy on the gateway, what users are allowed to run what commands (and where – against what Geneos data items). In using authentication with the REST API, a username and password are supplied as part of the HTTP request and are used to verify against users allowed to run commands on the gateway. Error messages for this are obvious:
"stderr": "User cannot be authenticated"
Note – gateway authentication over the REST API needs to be password based. As there is no Active Console the setting ‘allow system auth’ for pulling the username from the windows OS is not a viable authentication mechanism for the API.
The documentation section pertinent to this functionality is found here: Gateway Authentication Technical Reference.
Turning on authentication - This is the same as for other gateway authentication configurations. There is a setting in the configuration for turning Gateway Authentication is on (‘Authenticate Users’ is checked in Authentication section of gateway configuration). I’ve also enabled ‘case insensitive usernames’ (This helps prevent username case errors and red-herring access denials).
Request blocked - Having these settings enabled, a request to the REST API, even a simple one that only asks to list available commands, will be blocked.
The request:
GET – sent to http://[host]:[port]/rest/commands/all
(this is one of the most basic requests, which GETs a list of all the commands available that the gateway knows about).
Sending this GET request to that address will receive an error response:
"stderr": "User cannot be authenticated"
To remedy this we can configure authentication into the REST request. To do so we have to add an authenticated user definition to the gateway.
Adding a user to the gateway - In the gateway configuration, in authentication, we add a new user for our app (or testing) to use for its HTTP requests to the REST interface. My changes included
Adding a user definition:
Username: restUser
Password: password
I checked the setting “allow password auth” on the advanced tab for this user. This allows authenticating via a username, password pairing (needed for REST API).
I also added to this user various permissions for various commands. In particular for my example I added ‘execute’ level permissions to /* commands (all commands) on * targets (any Geneos data item target).
You could tailor these permission settings as needed and appropriate for your environment, restricting permissions to certain commands on certain targets, as granularly as desired, as would be done during regular Geneos authentication configuration. For more information on authentication and commands permissions, see the ITRS Authentication documentation here: Gateway Authentication (Command permissions).
After adding this entry, we send the same request but with authorization (of type “Basic Auth”) added, with the username and password being the ones specified in the user’s definition in the gateway authentication section.
Sending the request with these addenda, we find the command runs without issue.
At this point we have restrictive control over what commands can be run, at what locations in the Geneos space, by the user we’ve setup. Further, the REST API (or the gateway altogether) is only going to run commands by authenticated users, and the REST API is using the user we’ve configured for it, giving full restrictive granularity over what and how commands can be run.
Switching to our second topic, Geneos secure communications…
Secure Connections - In recent versions of Geneos (the last year or two), secure communications between components have been available. Connections between the probe, gateway, active console, webserver, and other geneos components can be encrypted with SSL certificates, protecting the data transferred.
This is true as well for the REST API, as another open connection the Gateway will receive information on this too can be encrypted. Indeed, one of the first steps we took in the prior blogs in this series was to ‘enable Insecure Comms’ for the Rest API itself, to simplify the requirements and configuration for those examples.
This section looks to remove that simplification.
Given commands can do quite a bit, authentication for command execution is important. As such, working with SSL here will compound on all the other configuration we’ve done, and work in conjunction with Authentication, to show encrypted, authenticated communications against the Gateway REST API.
The documentation section pertinent to this functionality is here: Secure Communications.
Build the Certificates - For SSL to work we need to build the certificates and key files to use in the communications. This is well documented in the ITRS documentation already, and examples are provided with terminal commands for how to create such files. For many users however, their organization will have its own Certificate Authority and internal procedures that should be followed for creating such certificates.
Steps for creating certs of an Unsigned, local, (and likely test use) are outlined here: Secure Communications (Example).
Provide Certificates to Gateway - However we get them, we should now have a gw.pem file containing the (public) certificate and (private) associated key. These will be provided to the gateway to enable its secure communications functions. This process is also documented in the ITRS documentation notably here: Creating Secure Geneos Servers.
The gateway is to be started up with the flag added to its process launch: “-ssl-certificate gw.pem”. Having added this, my gateway launch command looks like:
/opt/itrs/packages/gateway/active_prod/gateway2.linux_64 myGateway 54100 -setup /opt/itrs/gateway/gateways/poc_gateway/gateway.setup.xml -log /opt/itrs/gateway/gateways/poc_gateway/gateway.log -resources-dir /opt/itrs/packages/gateway/active_prod/resources -licd-port 7037 -licd-host localhost -ssl-certificate ~/gw.pem
(For me this is coming out of using the geneos-utils template for starting, stopping, and generally managing gateways, and other geneos components),
Enable Secure Communications - Having done this, and given the gateway the cipher for secure comms, we can enable them in the gateway configuration. This is done in the “Operating Environment” section of the gateway configuration.
When I first went to look at this, the gateway, by default, uses the deprecated listening port setting and older XML, your experience may be the same:
I switched to the new XML for my configuration:
This completed, the gateway should now be listening for secure communications on that port, and using the certificate we’ve provided.
REST API ‘insecure comms’ - To be extra sure our REST requests aren’t being allowed on insecure communication channels, we know that we’re using a different port, but even further we can remove that ‘allow insecure comms’ setting mentioned earlier, set on the REST API configuration itself.
Having done both of these, we’re ready to try to send our first request. Again, I’m doing this in Postman for simplicity. Also, I’m doing the simple GET Scenario outlined in the other blogs, which lists all available commands. Different and more complex scenarios are outlined above as well as in the other REST blogs, #1 “Rest API – More than a Snooze!” and #2 “Rest API – Rest Easy”. This simple scenario will work for showcasing our connectivity functionality.
Not surprisingly for our first try, we see we’re getting blocked. There are two things we need to do to get Postman to successfully play ball with the Gateways security here:
One is to change the request address to https, which, understandably, indicates to postman that we’re expecting secure or encrypted communications from this URL.
The second is by virtue of my using an unsigned certificate. Not all clients might have this issue (depending on how the cert was created, and the chain above it of what Certificate Authorities have endorsed it). For those of us with unsigned certificates, we need to tell Postman not to validate certificates, which is done in settings.
After clicking on the wrench, a pop-over menu appears, and the first option is ‘Settings.’ Clicking on this will open a settings window.
One of the first settings we see available to us, right on the first tab ‘General’ is a “SSL certificate validation.” Which indicates to postman to check certificates against authorities and make sure they’re from trusted sources. As we’ve created ours unsigned as a local encryption test, it won’t past that level of validation. We turn this setting off, at which point we can resend our request.
And we receive an authentication failed message. A step closer, but we forgot to include the authentication details we configured in the first section on authentication. Adding authentication information back onto the request and we find success:
At this point we have a request being sent encrypted over secure channels, with an authenticated user being used to restrict commands permissions and ensure requests are from authorized sources.
We believe the data transferred is being encrypted, but how can we be sure? To verify that the communications are indeed obfuscated, we can snoop the traffic that has been sent. There is a wide variety of ways to do this; I used wireshark (a free app), for simplicity and ease of use.
I re-enabled the insecure communications for the REST API, so that I could do an insecure request and a secure request and look at both, comparing the traffic between the two.
The Insecure port on my gateway is 54100, so the requests to (and from), that port should be in the clear. Here we see TCP traffic for that request:
And indeed we can see the response content is easily visible. Indeed, it matches exactly the beginning few lines of one of our usual responses, which we see in our Postman request when doing the GET request to get all commands (see above).
My gateway is configured with the secure options, as per above, and the secure port is at 54107. As such, the request to and response from this port should be encrypted and indecipherable, and that is indeed what we see.
It may seem a bit overkill to encrypt the traffic in this example, here, our request is only a GET request listing available commands. However, one can easily imagine a command returning sensitive data (imagine a view log file command returning contents from a sensitive log?). Even more immediately tangible, being able to see what commands are available could aid those with malicious intents, especially as some powerful and created commands may have conspicuous names. If I were up to nefarious ends and saw a command called ‘reset Market Data’ or “stop all processes,” they practically cry out for being triggered.
Further, it completely obfuscates the request being made, and the authorization details being used, which keeps them nice and secret (at least from snoopers).
And there we have it, the REST API using secure communications and Authentication in conjunction, to lock down, disguise, and restrict access to what commands can be run against the gateway and who can see the data transmitted.
For readers interested in reading further the topics discussed in this publication, the documentation for each topic is listed below:
