REST API – Don’t Lose Sleep!

A compendium of common pitfalls for the Geneos Commands Rest API, which allows automated, programmatic remediation actions in your monitoring.

As a Sales Engineer and Solutions Architect, I’ve had many occasions to work with the Gateway Command Rest API. In so doing, there have been moments of frustration, when I find myself repeatedly stumbling. 

I thought I’d compile a bunch of frequent errors and common pitfalls, making a common problems list, which I hope saves others the headache. I know I always find it helpful when I can find a digital writeup of someone having the same error. This writing synergizes well with previous blog – “REST API – More than a Snooze!”, or any other REST API use cases. 

For organization and simplicity, the topic of Authentication and common challenges around that was sectioned off into another blog and grouped alongside using secure communications with the REST API in the follow up blog – “REST API – Locking it Down.” 

A big help in all this, has been the API and HTTP request tool – Postman. It allows one to create one-off HTTP requests for testing purposes, see the responses you get back, save your built requests, and much more, all in a nice easy to use GUI. I’ll be using it throughout this publication. 

Without further ado, some common problems one may run across in working with the REST API…

Wrong gateway version   -   I’ve seen a lot clients run afoul of this one, and I myself have made the goof. Remember the Gateway Command REST API is only available as of Geneos GA 3.7. When sending Rest API commands at older gateways (such as GA3.5) one might see a message “404 Failed to service HTML Request”, followed by the URL on the Gateway you sent the request to.   

This is also the error one would see if the hostname or port are incorrect (AKA requests are being sent to the wrong place, not to a gateway), as well as if there were a network issue blocking requests.  

Wrong Hostname    -   If you send the request to a location without a gateway to receive it, naturally you won’t get a response. Depending on the client, this will usually result in a ‘No Response received’ error of some fashion, depending on how your HTTP client handles sending a request without receiving any response. 

This is different from a 404, as literally nothing is responding at that location, whereas 404 is a response code from a service that is there and listening, but was unable to perform the request (no such URL, permissions invalid, etc). 

Make sure your hostname is correct, or perhaps try using an IP address to bypass DNS mapping problems. 

Gateway REST Service is not enabled   -    The Rest service is not turned on in the gateway by default, so we have to turn it on. Similar to the prior scenario, if the gateway configuration is not set to turn on the REST service, then the gateway is not listening on that port. (Just as when the gateway is too old of a version to know about the REST API). 

If we don’t turn it on, the resulting error message is pretty direct:

{
   "error": "Rest Service not enabled"
}

To turn on the Rest Service, we make a configuration change in the gateway setup. The Rest settings are listed on the advanced tab of the Commands section of the configuration. Most important is Enabling the service. For ease of this example, the insecure communications setting is enabled as well; I will address working with secure communications, as well as authentication setup, in a subsequent post.

Same request, with setting change applied and Service enabled: 

This request, as detailed in the first blog of this series, is the simple ‘GET’ – List all commands request. 

Information about Status codes can be found here: Gateway Commands (HTTP Status codes)

Malformed JSON    -   This is the most common issue, and is easily remedied. Use a validator! There are website checkers that have richer and more robust interfaces, but even postman checks the JSON object in its interface. The error message for malformed JSON is relatively obvious - ‘JSON Command request malformed.’

Fortunately, Postman has an indicator for syntax errors in JSON requests, so create a test JSON package, and give it a run through Postman to see if it’s valid. Even better, use a web validator (although this isn’t always an option due to web access restrictions and security constraints).

An example escaped request, including an XPath is below (note the \” escaping of “s in the XPATH):

{  
  "command": "/RULES:showRules",
  "target": "/Geneos/gateway/directory/probe/managedEntity[(@name=\"app1ME\")]/sampler[(@name=\"Gateway Processes\")][(@type=\"\")]/dataview[(@name=\"Gateway Processes\")]/rows/row[(@name=\"gateway\")]/cell[(@column=\"instanceCount\")]" ,
  "args" : { "1" : "/Geneos/client/features/@RuleShowEx" }
}

Multi-targeted XPATHS    -   Currently the documentation specifies that the Rest API only accepts XPATHs that evaluate to one and only one Geneos item. The error returned is pretty conspicuous on this one, though may be unobvious if one isn’t aware of this constraint: 

  "stderr": "Command target matches more than one item"

There is a utility for de-evaluating a multi-target xpath into a list of single target xpaths, as part of the REST API. More on that to follow, but its use is specified in the documentation: Gateway Reference (Find items matching a target).

By default, as a safety mechanism, it will only evaluate to the first 100 matching objects, though the request can specify overriding this limit. This prevents a request accidentally evaluating to a massive number of objects and tying up the gateway, unless the user really means to. 

Wrong XPATH target for Command     -    Issuing a command with an XPath that identifies a Geneos object for which the command doesn’t apply will result in a fairly conspicuous error. Remember not all commands take any kind of Geneos object. For example, show rules doesn’t make any sense on a Managed Entity, and one wouldn’t ‘view Netprobe log’ on a gateway object. This could also apply for custom created commands, if they have a restricted scope and only apply in certain locations, and the requested XPath specifies a location outside those.

The Gateway has a special error message for this:

"stderr": "Command target is not applicable to the command"

In the below screenshot you can see the definition of the command testCommand only targets Managed Entities or individual dataview Cells, and only applies to these data item types.

Running it with a path that ends on a sampler (not a valid target for the command) yields the aforementioned error.

Running it on a path yielding a data item pertinent to that command does yield the expected result.

A utility that seems helpful for this, there is a REST request option for testing an xpath to see if it has data items which are targets for a given command. This may be tested in another blog, but it’s functionality is outlined here: Gateway Commands (Items which are targets for a given command).

Wrong arguments to command   -   Commands often require arguments, and in many cases not providing required arguments will naturally cause the command to fail. For some commands, the arguments required may seem arbitrary. Sometimes there are hidden or implicit arguments the command may require arguments that users aren’t used to or are unaware of. These error messages can be opaque and confusing, and can become a pitfall. Additionally, many arguments have defaults, or are static, and are optional. 

Documentation on Rest Service running Commands with Arguments.

Further, the documentation section on commands, their arguments and default values for arguments.

As mentioned before, some commands specify arguments that have a default, but which are not optional. This may be a fluke and only occurs very rarely, but to show an example of one and how it can be handled, we look at showRules. The documentation is here on Rule commands.

In trying to run the ‘showRules’ internal command, when specifying the command and the target (what else would be required, right?)

The request of:

{
  "command": "/RULES:showRules",
  "target": "/Geneos/gateway[(@name=\"gabrielTestGateway\")]/directory/probe[(@name=\"GabrielTestProbe\")]/managedEntity[(@name=\"app2ME\")]/sampler[(@name=\"Tester\")]/dataview[(@name=\"Tester\")]/rows/row[(@name=\"row1\")]/cell[(@column=\"col1\")]"
}

Will result in a response containing  -  “   "stderr": "Unable to evalute XPath specified in argument 1"   ”

We didn’t provide any arguments to our request. So, we take a step back, and go look at the command definition to see what arguments it may need or use.

The command specs can be found in the ‘Internal Commands’ section of the ITRS documentation.

Alternatively, one can always ask the REST API what commands are available, and it provides details about them. This is done by sending a GET request to the http://[hostname]:[portNum] /rest/commands/all URL. (see other sections of this blog, or blog #1).

From the long full list of commands that is returned by that request, the listing for showRules:

And we notice that it specifies one argument: 

{
  "name": "/RULES:showRules",
  "menuPath": [
    "Show Rules"
  ],
  "groups": [
    "/GATEWAY",
    "/RULES"
  ],
  "description": "Get information about the rules",
  "type": "Internal",
  "targets": [
    "/Geneos/gateway/directory/probe/managedEntity/sampler/dataview//cell",
    "/Geneos/gateway/directory/probe/managedEntity/sampler/dataview",
    "/Geneos/gateway/directory/probe/managedEntity/sampler",
    "/Geneos/gateway/directory/probe/managedEntity",
    "/Geneos/gateway/directory/probe",
    "/Geneos/gateway/directory"
  ],
  "arguments": [
    {
      "index": 1,
      "type": "XPath",
      "default": "/Geneos/client/features/@RuleShowEx"
    }
  ],
  "mimeTypes": [
    {
      "execLog": "text/plain"
    },
    {
      "stdout": "text/html"
    },
    {
      "stderr": "text/plain"
    }
  ]
}

So, going back to our request, we try it with an added args section, and specify for argument ‘1’ this quirky little XPath. 

Request:

{
  "command": "/RULES:showRules",
  "target": "/Geneos/gateway[(@name=\"gabrielTestGateway\")]/directory/probe[(@name=\"GabrielTestProbe\")]/managedEntity[(@name=\"app2ME\")]/sampler[(@name=\"Tester\")]/dataview[(@name=\"Tester\")]/rows/row[(@name=\"row1\")]/cell[(@column=\"col1\")]",
  "args": {
    "1": "/Geneos/client/features/@RuleShowEx"
  }
}

Turns out we’ll get our result, it returns the rule info (even if it is a bit hard to read). 

Implicit arguments seem to only happen for Internal Commands (this is somewhat understandable, as the active console probably provides these inputs when calling them).  

For internal commands, the full command definition and spec will indicate what arguments are available. The documentation doesn’t explicitly state which arguments are required and which are optional, but it at least provides info on what to experiment with.  Some, but not all, arguments have defaults.

Many, even most, arguments with defaults are optional. Consistently optional, it seems, are arguments of type ‘static’, as well as arguments with Type ‘User Input  *’. The SNOOZE:manualAllME command runs without any arguments, but has 5 specified in the documentation:

We see the request was successful and the command ran.

As a rule, follow the error messages, and start simple and work your way up. Also, use the show commands function (and/or the command documentation) to check a given command’s formal definition and what arguments it may need, then provide them in your requests.

Overall, there are a couple places one can get caught (as with anything), but the REST service gives pretty informative error messages, and is generally pretty easy to work with. It offers really nice functionality for triggering commands remotely and programmatically, and can be used for some nice integrations.

For further information about the REST API go here: Gateway Reference (REST Service).

For further information about Commands, their definitions, arguments, etc. see the documentation here: Gateway Commands (Internal Commands).