# Configure Custom Graphistry Ontology

The Graphistry ontology is a set of mappings that guide automatic visualization and simplify data worklows. This document overviews the ontology and how to extend. For the formal format, see Graphistry's [convict-format specification](https://github.com/graphistry/graphistry-cli/blob/master/docs/configure-investigation.md#schema).

## Out-of-the-box ontologies

Graphistry supports out-of-the-box ontologies of common systems:

* SIEM Models: Much of Splunk CIM, ArcSight CEF, and ElasticSearch Common Schema
* Vendor-specific models, such as FireEye HX/NX/iSIGHT, AWS CloudWatch, and WinLog
* Classifying IPs as internal vs. external by RFC 1918

See below for the list of built-in types they map to.

## Define custom ontologies

1. Edit `data/investigations/config/config.json` as per below
2. Restart docker service `pivot`: `./graphistry restart pivot`

Generally, you can limit the amount of work by mapping custom column names to built-in types, and thereby reuse their preconfigured settings.

## Ontology types

### Primary

Key `ontology` defines:

* For each type, such as `user`:
  * Default icon: string name supported by [Font Awesome 4](https://fontawesome.com/v4.7.0/icons/), such as [user-o](https://fontawesome.com/v4.7.0/icon/user-o)
  * Default color: string hex value, such as `#F00` for red
  * Default size: number, typically between 10 and 200
  * Displayed title: prioritized cascade based on entity type and available column names
* New types:
For the automatic table -> graph transform (aka hypergraph transform), the mapping from _table column names_ to _node entity types_.

### Secondary

Additional settings exist such specific to individual layouts and connectors

  
## How to extend the ontology

### Easiest: Ask Graphistry to do it for you! 

Ideally, you can provide representative sample data that has the columns and values of interest, and if a data schema is available, that too.

Ex: For Splunk users wanting support for a new product, provide the output of 
* `search index=some_product | fields * | dedup 20 event_type | head 1000`
* select all columns in the Field Selector
* download the CSV

### Add new types

For example, to create a new node type `ip`, 

1. Extend `data/investigations/config/config.json`:

```json
{
  "ontology": {
      "icons": {
         "ip": "device",
      },
      "colors": {
          "ip": "#F00",
      },
      "sizes": {
          "ip": 100
      }
  }
}   
```

2. Restart the pivot service:

```user@server.com:/var/graphistry $ ./graphistry stop pivot nginx && ./graphistry up -d```

### Override default node/edge titles

Graphistry picks the displayed title for each node and edge through the first match on the following cascade:

* By type match: Does the element's `type` value have a corresponding `byType` binding?
* By field match: Does the element contain a column name in `byField`?
* By `pointTitle` , `edgeTitle`, if available
* Use an element ID provided with the graph
* Use an element ID generated by the system

Ex:

```json
{
  "ontology": {
      "titles": {
          "byType": {
              "geo": "address",
              "user": "name"
          },
          "byField": ["src_ip", "dest_ip"]
      }
  }
}
```


### Configure new columns / new hypergraph transforms

The existing ontology may already have all the types you want, but a new data source may have columns that need to be mapped into it.

For example, to recognize `src_ip` and `dest_ip` columns as both generating `ip`-type nodes:

1. Extend `data/investigations/config/config.json`:

```json
{
   "ontology": {
       "products": [
           {
               "name": "my_extension_1",
               "colTypes": {
                   "src_ip": "ip",
                    "dest_ip": "ip"
               }
           }
       ]
   }
}
```

2. Restart the pivot service:
```
user@server.com:/var/graphistry $ ./graphistry stop pivot nginx && ./graphistry up -d
```

## Built-in types
The current set of built-in types is below. Upon system start, Graphistry emits the list of `Ontology types` for your installed version. You can also add your own (see above).

We recommend using built-in types when possible. Each type comes with a built-in color, icon, size, and mappings from common data sources to it. This saves you work now, and as more connectors become supported and new features are added, you will automatically benefit from them in the future as well.

```json
[
    "actor", "agent", "alert", "amazon", "amex", "arn", "asn", "availabilityzone", 
    "baidu", "bucket", 
    "cidr", "city", "cloud", "cny", "code", "container", "continent", "cookie", "count", "country", 
    "direction", "discover", "domain", "domainReputation", 
    "email", "error", "eur", "event", "extension", 
    "facebook", "file", "filePath", "filepath", "flag", "flickr", 
    "gateway", "gbp", "geo", "github", "google", "googleplus", "group", 
    "hash", "hashReputation", "host", "httpMethod", "httpmethod", 
    "id", "ils", "image", "inr", "instagram", "instance", "ip", "ipReputation", 
    "jcb", "jpy", 
    "key", "krw", 
    "language", "linkedin", "log", 
    "mac", "machine_type", "machinetype", "mastercard", "medium", "message", "money", 
    "name", "netbios", "networkinterface", "number", 
    "organization", "os", 
    "packer", "path", "payload", "paypal", "phone", "pinterest", "pod", "port", "process", "program", "protocol", 
    "qq", "quora", 
    "reddit", "role", "rub", 
    "score", "size", "skype", "slack", "snapchat", "state", "stripe", "subnet", 
    "tag", "telegram", "time", "timezone", "toolkit", "try", "tumblr", "twitch", "twitter", 
    "uri", "urifragment", "uripath", "uriquery", "url", "urlReputation", "usd", "user", "useragent", 
    "vendor", "version", "vine", "visa", "volume", "vpc", 
    "wechat", "weibo", "whatsapp", 
    "xbt", 
    "yahoo", "youtube"
]
```

## Layouts and IPs

Layouts have additional options. The most common to modify is to flag values for being "inside" in the network map layout.

You can put any regular expression here:

```json

  "layouts": {
    "network": {
        "ipInternalAcceptList": ["/10\.*/", "/127.0.0.1/"]
    }
  },

```


## Testing your ontology

1. Syntax errors: 

Graphistry tries to detect syntax error, and upon one, logs the error and stops. To see what is going on:

`docker ps` <- see if `pivot` is unhealthy or in a restart loop
`./graphistry logs pivot` <- see the precise error message

2. Satisfactory configuration

We recommend creating a Manual Data pivot. For example, to test various `ip` columns, use the following:

* Query: `[ {"src_ip": "10.10.0.0", "dest_ip": "10.10.0.1", "ip": "10.10.0.2"} ]`
* JQ: `.`
* Nodes: `src_ip`, `dest_ip`, `ip`