Configure Custom Graphistry Ontology

Configure Custom Graphistry Ontology#

The Graphistry ontology is a set of mappings that guide automatic visualization and simplify data worklows. This document overviews the ontology and how to extend. For the formal format, see Graphistry’s convict-format specification.

Out-of-the-box ontologies#

Graphistry supports out-of-the-box ontologies of common systems:

  • SIEM Models: Much of Splunk CIM, ArcSight CEF, and ElasticSearch Common Schema

  • Vendor-specific models, such as FireEye HX/NX/iSIGHT, AWS CloudWatch, and WinLog

  • Classifying IPs as internal vs. external by RFC 1918

See below for the list of built-in types they map to.

Define custom ontologies#

  1. Edit data/investigations/config/config.json as per below

  2. Restart docker service pivot: ./graphistry restart pivot

Generally, you can limit the amount of work by mapping custom column names to built-in types, and thereby reuse their preconfigured settings.

Ontology types#

Primary#

Key ontology defines:

  • For each type, such as user:

    • Default icon: string name supported by Font Awesome 4, such as user-o

    • Default color: string hex value, such as #F00 for red

    • Default size: number, typically between 10 and 200

    • Displayed title: prioritized cascade based on entity type and available column names

  • New types: For the automatic table -> graph transform (aka hypergraph transform), the mapping from table column names to node entity types.

Secondary#

Additional settings exist such specific to individual layouts and connectors

How to extend the ontology#

Easiest: Ask Graphistry to do it for you!#

Ideally, you can provide representative sample data that has the columns and values of interest, and if a data schema is available, that too.

Ex: For Splunk users wanting support for a new product, provide the output of

  • search index=some_product | fields * | dedup 20 event_type | head 1000

  • select all columns in the Field Selector

  • download the CSV

Add new types#

For example, to create a new node type ip,

  1. Extend data/investigations/config/config.json:

{
  "ontology": {
      "icons": {
         "ip": "device",
      },
      "colors": {
          "ip": "#F00",
      },
      "sizes": {
          "ip": 100
      }
  }
}

  1. Restart the pivot service:

user@server.com:/var/graphistry $ ./graphistry stop pivot nginx && ./graphistry up -d

Override default node/edge titles#

Graphistry picks the displayed title for each node and edge through the first match on the following cascade:

  • By type match: Does the element’s type value have a corresponding byType binding?

  • By field match: Does the element contain a column name in byField?

  • By pointTitle , edgeTitle, if available

  • Use an element ID provided with the graph

  • Use an element ID generated by the system

Ex:

{
  "ontology": {
      "titles": {
          "byType": {
              "geo": "address",
              "user": "name"
          },
          "byField": ["src_ip", "dest_ip"]
      }
  }
}

Configure new columns / new hypergraph transforms#

The existing ontology may already have all the types you want, but a new data source may have columns that need to be mapped into it.

For example, to recognize src_ip and dest_ip columns as both generating ip-type nodes:

  1. Extend data/investigations/config/config.json:

{
   "ontology": {
       "products": [
           {
               "name": "my_extension_1",
               "colTypes": {
                   "src_ip": "ip",
                    "dest_ip": "ip"
               }
           }
       ]
   }
}

  1. Restart the pivot service:

user@server.com:/var/graphistry $ ./graphistry stop pivot nginx && ./graphistry up -d

Built-in types#

The current set of built-in types is below. Upon system start, Graphistry emits the list of Ontology types for your installed version. You can also add your own (see above).

We recommend using built-in types when possible. Each type comes with a built-in color, icon, size, and mappings from common data sources to it. This saves you work now, and as more connectors become supported and new features are added, you will automatically benefit from them in the future as well.

[
    "actor", "agent", "alert", "amazon", "amex", "arn", "asn", "availabilityzone", 
    "baidu", "bucket", 
    "cidr", "city", "cloud", "cny", "code", "container", "continent", "cookie", "count", "country", 
    "direction", "discover", "domain", "domainReputation", 
    "email", "error", "eur", "event", "extension", 
    "facebook", "file", "filePath", "filepath", "flag", "flickr", 
    "gateway", "gbp", "geo", "github", "google", "googleplus", "group", 
    "hash", "hashReputation", "host", "httpMethod", "httpmethod", 
    "id", "ils", "image", "inr", "instagram", "instance", "ip", "ipReputation", 
    "jcb", "jpy", 
    "key", "krw", 
    "language", "linkedin", "log", 
    "mac", "machine_type", "machinetype", "mastercard", "medium", "message", "money", 
    "name", "netbios", "networkinterface", "number", 
    "organization", "os", 
    "packer", "path", "payload", "paypal", "phone", "pinterest", "pod", "port", "process", "program", "protocol", 
    "qq", "quora", 
    "reddit", "role", "rub", 
    "score", "size", "skype", "slack", "snapchat", "state", "stripe", "subnet", 
    "tag", "telegram", "time", "timezone", "toolkit", "try", "tumblr", "twitch", "twitter", 
    "uri", "urifragment", "uripath", "uriquery", "url", "urlReputation", "usd", "user", "useragent", 
    "vendor", "version", "vine", "visa", "volume", "vpc", 
    "wechat", "weibo", "whatsapp", 
    "xbt", 
    "yahoo", "youtube"
]

Layouts and IPs#

Layouts have additional options. The most common to modify is to flag values for being “inside” in the network map layout.

You can put any regular expression here:

  "layouts": {
    "network": {
        "ipInternalAcceptList": ["/10\.*/", "/127.0.0.1/"]
    }
  },

Testing your ontology#

  1. Syntax errors:

Graphistry tries to detect syntax error, and upon one, logs the error and stops. To see what is going on:

docker ps <- see if pivot is unhealthy or in a restart loop ./graphistry logs pivot <- see the precise error message

  1. Satisfactory configuration

We recommend creating a Manual Data pivot. For example, to test various ip columns, use the following:

  • Query: [ {"src_ip": "10.10.0.0", "dest_ip": "10.10.0.1", "ip": "10.10.0.2"} ]

  • JQ: .

  • Nodes: src_ip, dest_ip, ip