Git-for-data documentation

hoijnet · hoijnet · commit 39ada080a401 · 2025-05-15T21:30:55.000+02:00
diff --git a/src/app/docs/create-a-project-with-terminuscms/page.md b/src/app/docs/create-a-project-with-terminuscms/page.md
@@ -18,7 +18,7 @@ media:
 
 Follow this instructions to create a project/data product in DFRNT TerminusDB cloud -
 
-1.  Login to the dashboard at [dfrnt.com](https://dfrnt.com).
+1.  Login to the dashboard at [dfrnt.com](https://dfrnt.com/hypergraph-content-studio/).
 2.  Select the team that you want the project to live.
 3.  Choose 'New Data Product' from the top menu.
 4.  Provide your project ID - this can only contain alphanumeric characters and underscores.
diff --git a/src/app/docs/get-started/page.md b/src/app/docs/get-started/page.md
@@ -24,7 +24,7 @@ It is designed to give technical professionals and linked data professionals a s
 
 TerminusDB is open source and free to use, maintained by the team at DFRNT in collaboration with the community.
 
-Commercial hosting with a powerful modeller, visualisation, WOQL studio, and record editor user interface is available at [DFRNT.com](https://dfrnt.com) with a free trial. It works with both localhost open source TerminusDB and TerminusDB hosted in the cloud environments, including git-for-data synchronization of content repositories. 
+Commercial hosting with a powerful modeller, visualisation, WOQL studio, and record editor user interface is available at [DFRNT.com](https://dfrnt.com/hypergraph-content-studio/) with a free trial. It works with both localhost open source TerminusDB and TerminusDB hosted in the cloud environments, including git-for-data synchronization of content repositories. 
 
 When you sign up, you get full commercial hosting. This comes with generous limits to allow you to build without constraints, and then upgrade when needed.
 
diff --git a/src/app/docs/get-your-api-key-from-terminuscms/page.md b/src/app/docs/get-your-api-key-from-terminuscms/page.md
@@ -17,7 +17,7 @@ To use the Python or JavaScript client with TerminusDB, an API key is required.
 
 **1\. Log in**
 
-Log in to the modeller user interface [dfrnt.com](https://dfrnt.com)
+Log in to the modeller user interface [dfrnt.com](https://dfrnt.com/hypergraph-content-studio/)
 
 **2\. Select a Team**
 
diff --git a/src/app/docs/git-for-data-reference/page.md b/src/app/docs/git-for-data-reference/page.md
@@ -0,0 +1,90 @@
+---
+nextjs:
+  metadata:
+    title: Git-for-Data Reference
+    description: >-
+      A how-to guide showing the main operations related to Git-for-Data with TerminusDB in the cloud environment
+media: []
+---
+
+## Git-for-Data Reference
+
+Git-for-Data is a feature of TerminusDB that allows you to use git-like operations on TerminusDB data product branches, including the ability to transport data between TerminusDB instances using `clone`, `push`, `pull` and `fetch`.
+
+This is useful to collaborate on information by syncronizing content repositories with [cloud-hosted TerminusDB instances](https://dfrnt.com/hypergraph-content-studio/), and also between environments where network segmentation is strict, such as in IEC62443, Purdue model and other environments, as described in [Transfer data in operational technologies landscapes](/docs/operational-technologies-transfer). 
+
+In addition to the transportation of data between TerminusDB instances, Git-for-Data also allows you to manage data product branches using equivalents to `rebase` (replay commits onto a separate branch) and `merge` (apply) commands.
+
+The ability to `squash` a branch into a single commit, and to `reset` a branch to a previous commit complete the most important aspects of the git-for-data capabilities that enable model-based data to be managed effectively with revision control.
+
+Additionally, TerminusDB includes the ability to perform `diff` and `patch` operations on data product branches, which allow you to compare and apply specific changes between branches, much like git does with regular files, but instead of structured data.
+
+## The git-for-data operations
+
+Git-for-data operations use what is called `remotes`, references stored in the data product for remote data products, including branch information and which layers exist so that `push` and `pull` operations can compare the local state, with the remote state. 
+
+One or more `remotes` can be added to a data product.
+
+To read more about the git-like model itself in the TerminusDB explanation, see the [Git-like model](/docs/terminusdb-explanation/#git-like-model) section and the [Commit Graphs](/docs/graphs-explanation/#commit-graphs) section of the explanation of the TerminusDB graphs.
+
+### Fetching a Data Product
+
+You can fetch a data product by using the `fetch` command, which will retrieve information about the layers stored in a remote data product and update local references for the remote data product. Note that both `remotes` and `fetch` operate on a data product level.
+
+### Pull a Data Product
+
+You can pull a branch of a data product to a local branch by using the `pull` command. Missing layers layers in a remote data product branch will be transported and appended to the local branch that is pulled to as long as the history follows a straight revision control line and has not diverged.
+
+Schema operations are not pulled and need to be manually maintained. Only instance information is pulled between branches of data products.
+
+### Push a Data Product
+
+You can push a branch of a data product to a remote branch by using the `push` command. Missing layers layers in a local data product branch will be transported and appended to the remote branch that is pushed to as long as the history follows a straight revision control line and has not diverged.
+
+Schema operations are not pushed and need to be manually applied.
+
+## Cloning Data Products
+
+Data products contain a main branch and sometimes additional branches. You can clone a data product by using the `clone` command, which will create a copy of the data product in the same, or a different TerminusDB instance. 
+
+With cloning, the entire data product, including schema, all layers, all branches, title and description, and more, will be copied to the new data product in the new cloned data product. A remote is created automatically in the new data product, pointing to the original data product so that push and pull can be used easily.
+
+When cloning a branch, remote authorization information is included to let the two TerminusDB instances communicate with each other behind the scenes. Practically, the set of layers to that are included the the data product are calculated and then transferred back to the requesting TerminusDB instance.
+
+Cloning works well for moving a data product from a cloud-connected TerminusDB instance such as with the official DFRNT® [TerminusDB Git-for-Data Hosting](https://dfrnt.com/hypergraph-content-studio/), to another cloud instance, or to a local TerminusDB instance that can connect to the cloud instance.
+
+When the source TerminusDB instance is in a location that a cloud instance can't connect to, it becomes necessary to perform reverse cloning operations, which is described in the next section.
+
+### Reverse Branch Cloning
+
+When a data product is to be cloned from a local instance to a cloud instance, the `clone` command can't be used as the cloud instance can't connect to a TerminusDB that is technically not accessible from the cloud network.
+
+Instead, the set of layers to that are to be included has to be moved some other way, where the `fetch` and `push` commands are used.
+
+An example of how to use git-for-data is in how to move a data product from a local TerminusDB instance, to a cloud instance, is described in [Manual Reverse Branch Cloning](/docs/manual-reverse-branch-cloning).
+
+## Git-for-data branch operations
+
+### Rebase
+
+The `rebase` operation replays commits, layer changes, from one branch onto a different branch. This is similar to the `rebase` operation in git. The two branches must have a commit ancestor layer. The commit messages are retained.
+
+### Merge
+
+The `merge` operation applies commits, layer changes, from one branch onto a different branch as one squashed commit, where all changes are combined into one commit. This is similar to the `merge` operation in git. The two branches must have a commit ancestor layer.
+
+### Squash
+
+The `squash` operation merges all commits in the same branch into a single commit with flat history.
+
+### Reset
+
+The `reset` operation resets a branch to a previous commit, discarding all commits after the specified commit.
+
+### Diff and patch
+
+`diff` and `patch` operations are available to compare and apply specific changes between branches of data products. These operations are described in [JSON Diff and Patch](/docs/json-diff-and-patch).
+
+## More about git-for-data
+
+Read more about how git-for-data can be used in operational technologies environments in [Transfer data in operational technologies landscapes](/docs/operational-technologies-transfer), and how to clone individual branches of data products in [Manual Reverse Branch Cloning](/docs/manual-reverse-branch-cloning).
diff --git a/src/app/docs/how-to-connect-terminuscms/page.md b/src/app/docs/how-to-connect-terminuscms/page.md
@@ -14,7 +14,7 @@ media:
     value: https://cdn.terminusdb.com/img/terminus-cms/client_api_connection.jpeg
 ---
 
-Cloud hosted TemrinusDB is available from [https://dfrnt.com](https://dfrnt.com). Register an account or sign up if you haven't already to get started fast.
+Cloud hosted TemrinusDB is available from [https://dfrnt.com](https://dfrnt.com/hypergraph-content-studio/). Register an account or sign up if you haven't already to get started fast.
 
 Click on the **team** that you wish to use or create a new one. A default one will be already created for you. After selecting your team, click on your profile icon in the upper-right corner and click on `Profile`.
 
diff --git a/src/app/docs/manual-reverse-branch-cloning/page.md b/src/app/docs/manual-reverse-branch-cloning/page.md
@@ -0,0 +1,24 @@
+---
+nextjs:
+  metadata:
+    title: Manual reverse branch cloning
+    description: How to manually clone branches between data products 
+media: []
+---
+
+## Reverse Branch Cloning
+
+Reverse branch clonings is useful when the basic clone command can't be used, such as when the source TerminusDB instance is in a location that a cloud instance can't connect to. Another reason could be that just a branch should be moved between two data products and not the entire data product.
+
+In most circumstances, the `clone` command can be used to move a data product branch from a cloud TerminusDB instance to a localhost instance, or from a cloud instance to another cloud instance or team.
+
+How to reverse clone a data product branch from a localhost instance to a cloud instance, is described here. The baseic procedure goes like this:
+
+1. If the cloud data product does not exist, create a new empty one
+1. Setup a remote in the localhost data product from the cloud instance
+1. Fetch the remote data product from the cloud instance
+1. Push the relevant branch, such as `main` from localhost to the remote data product `main` branch
+1. Move any additional branches that should be moved
+1. Copy title, description and other attributes of the data product to the cloud version. 
+
+Read more about [git-for-data](/docs/git-for-data-reference) for more information on how to use TerminusDB and git-for-data.
diff --git a/src/app/docs/operational-technologies-transfer/page.md b/src/app/docs/operational-technologies-transfer/page.md
@@ -0,0 +1,19 @@
+---
+nextjs:
+  metadata:
+    title: Operational Technologies Information Transfer
+    description: How to use TerminusDB in Operational Technologies environments for IEC62443, Purdue model and other environments where network segmentation is strict.
+media: []
+---
+
+## Transfer data in operational technologies landscapes
+
+TerminusDB is designed for many different kinds of environments, including cloud, local, firewalled, DMZ and airgapped environments. The git-for-data features enables knowledge graphs stored in TerminusDB to be transferred between environments that have special communications requirements.
+
+In IEC62443, purdue network and other environments, communications are only allowed to move outwards in an onion-layered model, where the outermost layer of the operational technology environment is a DMZ that only allows inbound communications both from an IT environment and the OT environments, preventing all outbound communications.
+
+TerminusDB is well positioned for data transports in such environments thanks to the bidirectional `push` and `pull` protocols to transport the latest information as layer updates and where necessary also specific branches.
+
+An analytics data product can be pushed in layers (between TerminusDB instances) all the way to a DMZ network handoff point, and then `pull`ed from the DMZ into the IT environment. This prevents any outbound communication from the DMZ environment, whilst the information can flow as part of a digital twin of the operational environment.
+
+Read more about the git-for-data of knowledge graphs in the [Git-for-Data Reference](/docs/git-for-data-reference) page.
diff --git a/src/app/docs/page.md b/src/app/docs/page.md
@@ -25,7 +25,7 @@ It is designed to give technical professionals and linked data professionals a s
 
 TerminusDB is open source and free to use, maintained by the team at DFRNT in collaboration with the community. 
 
-Commercial hosting with a powerful modeller, visualisation, WOQL studio, and record editor user interface is available at [DFRNT.com](https://dfrnt.com) with a free trial. It works with both localhost open source TerminusDB and TerminusDB hosted in the cloud environments, including git-for-data synchronization of content repositories. 
+Commercial hosting with a powerful modeller, visualisation, WOQL studio, and record editor user interface is available at [DFRNT.com](https://dfrnt.com/hypergraph-content-studio/) with a free trial. It works with both localhost open source TerminusDB and TerminusDB hosted in the cloud environments, including git-for-data synchronization of content repositories. 
 
 When you sign up, you get full commercial hosting. This comes with generous limits to allow you to build without constraints, and then upgrade when needed.
 
diff --git a/src/app/docs/product-tour/page.md b/src/app/docs/product-tour/page.md
@@ -26,7 +26,7 @@ TerminusDB includes many features to build content infrastructures for complex e
 
 ## Sign Up and Try Out a Demo Project
 
-Sign up for DFRNT TerminusDB cloud for free at: [https://dfrnt.com](https://dfrnt.com).
+Sign up for DFRNT TerminusDB cloud for free at: [https://dfrnt.com](https://dfrnt.com/hypergraph-content-studio/).
 
 Verify your email address by clicking on the link emailed to you and logging in.
 
diff --git a/src/app/docs/terminusdb-internals/page.md b/src/app/docs/terminusdb-internals/page.md
@@ -0,0 +1,23 @@
+# TerminusDB Internals
+
+## TerminusDB Server
+
+### Command line Prolog query interface
+
+The TerminusDB server provides a command line datalog query interface. It does not fully follow the WOQL syntax, but is aligned to the internals of TerminusDB and thus offers a more direct way to query the database.
+
+The query interface is available as `./terminusdb query`. Variables are dynamically bound using the `v/1` predicate, see below. To get the responses as JSON, use the `--json` flag.
+
+```
+./terminusdb query admin/sandbox/local/branch/main --json 't(v(a),v(b),v(c))'
+```
+
+### Start in interactive mode
+To start the server in prolog interactive mode, use the `serve --interactive` flag. It will start the server on port 6363 and it enables runtime introspection and to change behaviour of the runtime engine. To be able to run commands, you need to initialize the environment and to load a specific database to operate on. This is beyond the scope of this tutorial.
+
+```
+./terminusdb serve --interactive
+```
+
+## Running queries
+
diff --git a/src/app/docs/woql-getting-started/page.md b/src/app/docs/woql-getting-started/page.md
@@ -22,7 +22,7 @@ This page provides examples of WOQL queries and explanations for how to use it i
 
 In the tutorial we will use the Javascript/Typescript dialect of WOQL. The logic is the same, but the syntax for Python follows traditional Python patterns. For details, refer to the syntax of each.
 
-All examples have been tested in the TerminusDB Logical Studio provided by [DFRNT](https://dfrnt.com). The reader will be expected to have already built a few first classes and stored documents in TerminusDB before embarking on the examples in the tutorial.
+All examples have been tested in the TerminusDB Logical Studio provided by [DFRNT](https://dfrnt.com/hypergraph-content-studio/). The reader will be expected to have already built a few first classes and stored documents in TerminusDB before embarking on the examples in the tutorial.
 
 ## Predicates, literals and variables in WOQL
 
diff --git a/src/app/page.md b/src/app/page.md
@@ -23,7 +23,7 @@ TerminusDB includes many features to build content infrastructures for complex e
 * Come visit us on [Discord](https://discord.gg/yTJKAma)
 * [GitHub](https://github.com/terminusdb/terminusdb)
 
-[DFRNT](https://dfrnt.com) took the lead of the continued development of TerminusDB in 2025 as main sponsors of the project, and community efforts. We offer cloud and managed service TerminusDB hosting, and an advanced data product modeller that can also be used with locally installed open source TerminusDB instances. We have been building our dedicated modeller since 2021.
+[DFRNT](https://dfrnt.com/hypergraph-content-studio/) took the lead of the continued development of TerminusDB in 2025 as main sponsors of the project, and community efforts. We offer cloud and managed service TerminusDB hosting, and an advanced data product modeller that can also be used with locally installed open source TerminusDB instances. We have been building our dedicated modeller since 2021.
 
 Our customers use TerminusDB for various complex information processing needs: such as Product Information Management, Reference Data Management, Cultural Heritage, special purpose Customer Relationship Management, Non-Profit Records Management, Linked Data, and more. 
 
@@ -35,7 +35,7 @@ We maintain close communications with the original TerminusDB development team w
 
 The easiest way to get started with the open source community version on your own computer is by [getting started with Docker](/docs/install-terminusdb-as-a-docker-container/).
 
-TerminusDB is open source and free to use. Commercial hosting with a powerful modeller, visualisation, WOQL studio, and record editor user interface is available at [DFRNT](https://dfrnt.com) with a free trial. 
+TerminusDB is open source and free to use. Commercial hosting with a powerful modeller, visualisation, WOQL studio, and record editor user interface is available at [DFRNT](https://dfrnt.com/hypergraph-content-studio/) with a free trial. 
 
 The commercial modeller, visualisations and records management tool works with both localhost open source TerminusDB and TerminusDB hosted in the cloud environments, including git-for-data synchronization of content repositories between cloud and localhost.
 
diff --git a/src/lib/navigation.ts b/src/lib/navigation.ts
@@ -477,6 +477,20 @@ export const navigation: Navigation[] = [
         title: 'Python Client Reference',
         href: '/docs/python',
       },
+      {
+        title: 'Git-for-Data Reference',
+        href: '/docs/git-for-data-reference',
+        links: [
+          {
+            title: 'Manual reverse branch cloning',
+            href: '/docs/manual-reverse-branch-cloning',
+          },
+          {
+            title: 'Strictly segmented environments',
+            href: '/docs/operational-technologies-transfer',
+          },
+        ],
+      },
       {
         title: 'GraphQL Reference',
         href: '/docs/graphql-query-reference',
