Release notes for 2.37.40

Overview:

Much of the improvements in this release are the internal foundations for upcoming features around collaboration, Hub, and new UI modes. User visible improvements are primarily around sharing, GPU/RAPIDS-ready JupyterLabs, performance, and fixes.

Highlights

• Features
  • Sharing panel
  • Jupyter Lab
  • GPU Monitoring Dashboard
  • Gremlin integration: TinkerPop, AWS Neptune, Azure Cosmos DB
• Infrastructure
  • Dependency upgrades
  • Override file
  • Automatic log rotation
  • Notebook resilience
  • Beginning of remote & cloud-native storage
  • Backups: DB load script
• Admin
  • Organizations
  • Constants
  • Roadmap
  • Nginx
• Fixes & tweaks
  
  • Nexus constants
  • Fewer 502s
  • Dask restarts
• Migration
  • Retired CUDA 10.2 in prep for RAPIDS 0.21

Graphistry Versions

Server	2.37.NEXT
JS React+Vanilla	3.7.6
Python PyGraphistry client	0.19.4 -> 0.20.1

Third-Party Versions

Caddy	2.3.0
CUDA (In-Docker)	11.0 (+ 10.2 deprecated)
Django	3.1
Docker (CE)	19.03.2
Docker Compose	1.24.1
Elasticsearch node driver	14.2.2
Gremlinpython	3.4.8
Jupyter	1.0.0
Neo4j-python-driver	4.1.0
NodeJS	14.6.1
Pandas	1.2
Postgres	12.5
Python	3.7
Neo4j node driver	4.1.1
Notebook	6.2.0
RAPIDS	0.19
Arrow	1.0.1
Splunk node SDK	1.9.0
Tornado	6.1

Features

• Sharing panel: Visualizations still default to public view / self-edit. You can now flip a visualization to private view, and invite users (identified by email address) with either view or view+edit access.  
• Jupyter Lab: The Jupyter environment has been upgraded from the minimal notebook interface to the labs studio one
• GPU Monitoring Dashboard: As part of the Jupyter Labs update, a GPU monitoring dashboard is preinstalled
• Dask Monitoring Dashboard: Also port of the Jupyter Labs update, a Dask task scheduler dashboard, including GPU-based dask-cudf tasks, has been added. Check `docker-compose logs -f -t --tail=100 dask-scheduler` for the location of Graphistry's built-in dask scheduler, and you can create new ones from the dashboard
• Gremlin Integration: TinkerPop, Neptune, Cosmos: PyGraphistry API and demo support

Infrastructure

• CUDA 10.2 Deprecated (Same as before): RAPIDS 0.19 is the last version of RAPIDS that supports CUDA < 11.0; RAPIDS 0.21 requires CUDA 11.0+ . The next release of Graphistry will require CUDA 11.0+.
• Override file: Stub `docker-compose.override.yml` is included to encourage editing that file instead of the provided `docker-compose.yml`
• Automatic log rotation: Each service now defaults to 10MB non-blocking log files before automatically rotating to a new one and compressing the old, up to 10 rotations old
  • Expected size: 910MB = 10 services x (9 rotations x 90% compression + 1) * 10MB current
  • Defined in `docker-compose.yml`; modify by creating a custom `docker-compose.override.yml`
  • BlazingSQL logs are reset every 1hr (working around upstream issue of not respecting log configurations)
• General system upgrades: See version list. Upgrades from React 16 -> React 17 may cause unexpected regressions in the UI; please report in case we missed any
• Significant database schema changes: In preparation for teams+orgs security features, schemas have been updated with granular tracking of ownership; standard migration should handle
• Casbin security policies: Dataset/file/session use is now increasingly governed by central Casbin policies stored in the database as part of the move to enabling user-defined access policies. This release aims to preserve existing behavior; please report in case we missed any regressions
• Internal JWT security: Internal services calls are increasingly tagged by JWT user context as we harden permissions (and expose deeper control of permissions)
• Nginx base image: Switched from Alpine to Debian. This is slightly faster and slightly bigger.
• Notebook resilience: Health monitor and restart now based on an HTTP GET success instead of process existence, which aids cases where notebooks are up but frozen
• Beginning of remote & cloud-native storage: Starting with data/{files,datasets} and for Azure, when a remote store is provided, the local disk will be used as a cache instead the store. Off by default; contact if a priority for your team. The direction is currently download-only; server->storage sync must be manually performed (e.g., az copy). To use, set:
  AZ_STORAGE_ACCOUNT_NAME=mystore
  AZ_STORAGE_ACCOUNT_KEY=mykey
  AZ_STORAGE_CONTAINER_NAME=mycontainer
• Backups: DB load script: See your etc/scripts folder for loading DB backups directly from a `backup.sql`

Admin

• Organizations: All users now implicitly belong to a personal Organization and potentially multiple other Organizations. This impacts using the admin panel for account creation: you must select an initial organization. Upcoming releases will make Organizations a user-visible feature and provide easy-to-use interfaces for user-level (non-admin) Organization creation, invitations, and team management.
• Constants: Fixes to the admin settings handling should improve start behavior
• Roadmap: Added public release roadmap tracker
• Nginx: Logs now exposed to Docker; set `NGINX_DEBUG=-debug` for more verbose

Fixes & tweaks

• Constants: The constant handling had issues stemming to the Django 3.1 upgrade and upstream bugs in plugins' support for it requiring workarounds
• Fewer 502s: Most (but not all) URLs giving occasional 502 errors even under light load have been fixed 
• Node IDs: Better handling of node bindings where node="id"; fixes cases where UI may have been stuck at "loading 15%"
• Dask restarts: HA workaround for Dask workers & Dask-using-services that occasionally deadlock during autohealing when initial termination involved a Lock

Migration

As noted in previous notes, this is the first release requiring CUDA 11.0+ driver compatibility. Nvidia RAPIDS 0.21 requires CUDA 11.0+, forcing this change from CUDA 10.2