Release notes for v2.31.7 – Graphistry

Highlights

Infrastructure: Caddy, Node, health checks, Docker subnet pinning, disabled managed logging by default
Core:
- AWS GovCloud: This will be the first release published to AWS GovCloud
- Dashboarding: StreamLit demos
- Page load speedups
- Connectors & integrations: Neo4j 4.X, StreamLit, Gremlin 3.4.8, Jupyter upgrade
- Improved date type inference: For /pivot connectors
API:
- Sessions: Expose experimental inspection endpoint in preparation for live streaming
Administration:
- Bootstrap scripts: For native Ubuntu 18.04 host environments
Documentation:
- Connectors, pivots, templates: Moving from GitHub to the main docs
- Base pivots: Added for simplifying custom pivot creation
Fixes! Core fix to uploads, histograms, & encodings for wider value ranges, built-in demos

Graphistry Versions

Server	2.31.NEXT
JS React+Vanilla	3.7.1
Python PyGraphistry client	0.11.7 (from 0.11.6)

Third-Party Versions

BlazingSQL	0.14
Caddy	1.0.3
CUDA (In-Docker)	10.0
Django	3.0
Docker (CE)	19.03.2
Docker Compose	1.24.1
Elasticsearch node driver	14.2.2
Gremlinpython	3.4.8
Jupyter	1.0.0
Neo4j-python-driver	4.1.0
NodeJS	12.X -> 14.X
Pandas	0.25.3
Postgres	12.3
Python	3.7.3 -> 3.7.7
Neo4j node driver	1.7.6 -> 4.1.1
Notebook	6.1.1
RAPIDS	0.14.0 -> 0.14.1
Arrow	0.15.0
Splunk node SDK	1.9.0
Tornado	5.1.1

Summary

At the highest-level, this release represents a massive set of backend dependency upgrades (let us know if you find stability issues!), support for Neo4j 4.X and secure StreamLit dashboarding, fixes around the filters and the 2.0 API, more health check support, and beginning to expose the sessions API in preparation for the live streaming API. Finally, for Marketplace users, we disabled managed logging by default.

New Features

Infrastructure
- Health checks:
- Added additional container service and subprocess checks (see full list)
- Additional health checks automatically run by docker-compose: Caddy, Redis, ...
API
- Sessions API endpoint: Initially for inspection of active sessions

Docs

StreamLit dashboarding demo app: See graph-app-kit on GitHub
Investigation template docs moved from Github to main (connectors, pivots, templates)
Base pivot list added to main docs, providing a reference for creating custom pivots that extend them
data/config/Caddyfile.default: Sample Caddyfile configurations
- HTTP, HTTPS with auto-TLS, HTTPS with custom certs
- StreamLit demo of reusing Caddy Graphistry JWT cookie auth for custom services
Sessions API (read)
Built-in visualization demos should work again (Les Mis, Facebook, ...)

Fixes & Tweaks

Core
- Histograms and encodings: Wide value ranges are now being correctly interpreted during histogram coloring and encodings based on them, e.g., when an attribute contains every number from 0 through 1000 and then used for coloring, or similar for time values
- Page load should be faster
- Uploads: Empty columns in 2.0 should now upload successfully (workaround for cudf #5898)
- Pivot: Improved inference for date cols (_at/time/date in col name, or values that match common formats or numeric ranges)
- Time zones: Especially around /pivot, changes reflect an approach of considering server-generated timestamps to be UTC/GMT+0, while client timestamps are in local time and get normalized to UTC as needed
Infrastructure
- Docker subnet (experimental): The configuration files now make the subnet for the Docker network (grph_net) an explicit environment variable, GRAPHISTRY_DOCKER_SUBNET, with default 172.21.0.0/16, and uses it as part of internal protections. (Planned deprecation in 2021 as part of retiring api=1.)
- Auth: JWT sessions persist across nexus service restarts when DJANGO_SECRET_KEY is fixed
- Node: Updated from Node 12.X (LTS) -> Node 14.X (LTS), including many infra dependencies (Fastify, Got, Socket, Pino, PM2, ...)
- Python 3.7.7: We unpinned Python minor version to just be "3.7.X" to more reliably follow latest RAPIDS ecosystem version
- Neo4j:
  - 4.X support: The BOLT driver for `/pivot` is updated to 4.1.1, which should still support Neo4j 3.X and add support for Neo4j 4.X
  - Path support: Cypher queries that return paths in `/pivot` should now work if they did not before
- Caddy:
  - Updated to 1.0.4 (EOL) in prep for move to 2.0
  - Plugins added for more powerful custom configurations, such as reusing Graphisty as a bastion server with built-in accounts. Will change with 2.0:
    
    dyndns,http.authz,http.awses,http.awslambda,http.cache,http.cors,http.datadog,http.expires,http.filter,http.forwardproxy,http.geoip,http.ipfilter,http.jwt,http.login,http.nobots,http.permission,http.prometheus,http.proxyprotocol,http.ratelimit,http.realip,http.reauth,http.recaptcha,http.supervisor,tls.dns.cloudflare,tls.dns.duckdns,tls.dns.dyn,tls.dns.godaddy,tls.dns.googlecloud,tls.dns.linode,tls.dns.namecheap,tls.dns.ns1,tls.dns.otc,tls.dns.ovh,tls.dns.powerdns,tls.dns.rackspace,tls.dns.route53

Administration

We are starting to expose API endpoints as scripts in `etc/scripts/rest`, starting with the session API (sessions.sh and session.sh)
Graphistry's Docker network grph_net is now by default pinned to subnet 172.21.0.0/16. Override it in custom.env using environment variable GRAPHISTRY_DOCKER_SUBNET.
Marketplace users (AWS, Azure) no longer have managed Graphistry logging enabled by default, see migration notes for enabling
Environment bootstrap scripts (Nvidia drivers, Docker, docker-compose, ...) are now available for Ubuntu 18.04 hosts in etc/scripts/bootstrap (Tested with GCP)

Migration

Highly recommended due to bug fixes

Regular migration over the previous release
- If you have subnet 172.21.0.0/16 prohibited for Graphistry use such as due to being used by another Docker service, modify custom.env to add GRAPHISTRY_DOCKER_SUBNET for another range
- For marketplace users, if you want to continue managed Graphistry logging vs. local json, copy `etc/assets/remote-splunk-logging.json` to your `/etc/docker/daemon.json` and restart docker (`sudo service docker restart`)
See migration notes for upgrading over earlier releases