IGNITE-21928 Describe Catalog operation flow in README (#3585)
diff --git a/modules/catalog/README.md b/modules/catalog/README.md
index 3097a07..0655689 100644
--- a/modules/catalog/README.md
+++ b/modules/catalog/README.md
@@ -1,7 +1,88 @@
# Catalog module
-This module provides database catalog service implementation and descriptor for catalog objects.
+Catalog is a component that responsible for managing descriptors of objects available in cluster.
+Additionally, it serves as access point for other components to obtain proper version of descriptors
+with respect to provided version of catalog or timestamp.
-See [CatalogService](src/main/java/org/apache/ignite/internal/catalog/CatalogServiceImpl.java)
+This module provides implementation of catalog service as well as internal API for accessing the
+objects descriptors.
-TODO: IGNITE-19082 Add comprehensive description.
\ No newline at end of file
+## Base components description
+
+* [CatalogObjectDescriptor](src/main/java/org/apache/ignite/internal/catalog/descriptors/CatalogObjectDescriptor.java) --
+ base class for objects managed by catalog, like tables, indexes, zones, etc.
+* [CatalogService](src/main/java/org/apache/ignite/internal/catalog/CatalogService.java) -- provides
+ methods to access catalog's objects' descriptors of the exact version and/or last actual version at
+ a given timestamp, which is a logical point-in-time. Besides, components may subscribe for catalog's
+ events to be notified as soon as the change of interest is happened.
+* [CatalogCommand](src/main/java/org/apache/ignite/internal/catalog/CatalogCommand.java) -- denotes
+ particular modification of the catalog, like creation of particular table, for example.
+* [CatalogManager](src/main/java/org/apache/ignite/internal/catalog/CatalogManager.java) -- provides
+ methods for object manipulation (like creation of new object and/or modification of existing ones),
+ also takes care of component lifecycle.
+* [UpdateEntry](src/main/java/org/apache/ignite/internal/catalog/storage/UpdateEntry.java) --
+ result of applying [CatalogCommand](src/main/java/org/apache/ignite/internal/catalog/CatalogCommand.java),
+ represents delta required to move current version of catalog to state defined in the given command.
+* [UpdateLog](src/main/java/org/apache/ignite/internal/catalog/storage/UpdateLog.java) -- distributed
+ log of incremental updates.
+
+## Guarantee
+
+For modify operation (invocation of `CatalogManager.execute()`), a resulting future will be completed
+as soon as version in which results of the command take place becomes available on every node of the
+cluster. This "availability" is determined as "version activation time plus some duration to make
+sure changes are propagated and applied within the cluster", where `some duration` defined by
+`schemaSync.delayDuration` configuration property. This implies, that consequent read access to
+catalog service with latest timestamp (`HybridClock.now()`) from any node will return the catalog of
+version that incorporates results of the command (assuming there were no concurrent updates overwriting
+modifications made by initial command). This also implies, that concurrent access to the catalog service
+may return catalog of version incorporating results of the command even if resulting future has not been
+yet resolved. Additionally, there is an extra await to make sure, that locally new version will
+_always_ be available immediately after resulting future is completed.
+
+For read access, it's up to the caller to make sure, that version of interest is available in local
+cache. Use `CatalogService.catalogReadyFuture` to wait for particular version, or
+`SchemaSyncService.waitForMetadataCompleteness` if you need a version which is active at provided
+point in time.
+
+Consumers of catalog's events are allowed to read catalog at version from event.
+
+## How it works
+
+### CatalogManager
+
+Every modification of catalog is split on two phases: accept and apply. During accept phase, given command
+is validated, then used to generate a list of update entries to save, and, finally, list is saved to
+distributed update log. Below is a sequence diagram describing accept phase:
+
+
+After update entries are saved to the log, it is the job of the update log to propagate updates across the
+cluster. On every node, update log notifies catalog manager about new update entries, and latter applies
+them and stores new version of a catalog in a local cache. Below is a sequence diagram describing apply phase:
+
+
+### UpdateLog
+
+Current implementation of update log based on a metastorage. Update entries of version N are stored by
+`catalog.update.{N}` key. Also, the latest known version is stored by `catalog.version` key. Updates
+are saved on CAS manner with condition `newVersion == value(catalog.version)`.
+
+#### Update log compaction
+
+Over time, the log may grow to a humongous size. To address this, snapshotting was introduced to UpdateLog.
+When saving snapshot of version N, update entries stored by `catalog.update.{N}` will be overwritten with
+catalog's snapshot of this version. Every update entries of version lower that version of snapshot will be
+removed. The earliest available version of catalog is tracked under `catalog.snapshot.version` key.
+
+#### Update log recovery
+
+During recovery, we read update entries one by one for all version starting with "earliest available" till
+version stored by `catalog.version` key, and apply those updates entries once again.
+
+#### Update log entries serialization
+
+Update log entries are serialized by custom marshallers (see
+[UpdateLogMarshaller](src/main/java/org/apache/ignite/internal/catalog/storage/serialization/UpdateLogMarshaller.java)
+and [MarshallableEntry](src/main/java/org/apache/ignite/internal/catalog/storage/serialization/MarshallableEntry.java)
+for details). At the moment, backward compatibility is preserved by increasing the version of the protocol,
+but more sophisticated approach may be introduced later.
\ No newline at end of file
diff --git a/modules/catalog/tech-notes/acceptUpdateFlow.puml b/modules/catalog/tech-notes/acceptUpdateFlow.puml
new file mode 100644
index 0000000..96e2500
--- /dev/null
+++ b/modules/catalog/tech-notes/acceptUpdateFlow.puml
@@ -0,0 +1,53 @@
+/'
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+'/
+
+@startuml
+title Accepting an update
+
+autonumber
+
+actor User
+
+activate User
+User -> CatalogManager ++ : Create table (CreateTableCommand)
+CatalogManager -> CatalogManager : Acquire latest catalog version
+CatalogManager -> CatalogManager : Calculates incremental updates by applying provided command
+CatalogManager -> UpdateLog -- : Saves diffs to update log under latestVersion+1
+UpdateLog --> CatalogManager ++
+CatalogManager -> CatalogManager --++ : Wait catalog of new version to appear in local cache
+
+alt updates were saved
+
+ CatalogManager -> CatalogManager --++ : Wait for delay duration
+ CatalogManager -> User -- : Success (Completes the future successfully)
+
+else updates were not saved because of concurrent update
+
+ alt retry limit is not exceeded
+
+ CatalogManager -> CatalogManager : Restart from p.2
+
+ else retry limit is exceeded
+
+ CatalogManager -> User -- : Failure (Completes future exceptionally)
+
+ end
+
+end
+
+deactivate User
+@enduml
\ No newline at end of file
diff --git a/modules/catalog/tech-notes/accept_update_flow.png b/modules/catalog/tech-notes/accept_update_flow.png
new file mode 100644
index 0000000..4868d5d
--- /dev/null
+++ b/modules/catalog/tech-notes/accept_update_flow.png
Binary files differ
diff --git a/modules/catalog/tech-notes/applyUpdateFlow.puml b/modules/catalog/tech-notes/applyUpdateFlow.puml
new file mode 100644
index 0000000..1f5402f
--- /dev/null
+++ b/modules/catalog/tech-notes/applyUpdateFlow.puml
@@ -0,0 +1,32 @@
+/'
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+'/
+
+@startuml
+title Applying an update
+
+activate UpdateLog
+
+UpdateLog --> CatalogManager ++: Notifies about new updates
+CatalogManager -> CatalogManager : Acquire catalog of version equal to version from update minus 1
+CatalogManager -> CatalogManager : Applies updates to the catalog
+CatalogManager -> CatalogManager : Registers new catalog in a local cache
+CatalogManager -> CatalogManager : Notifies other components about changes in catalog
+CatalogManager --> UpdateLog --
+
+deactivate UpdateLog
+
+@enduml
\ No newline at end of file
diff --git a/modules/catalog/tech-notes/apply_update_flow.png b/modules/catalog/tech-notes/apply_update_flow.png
new file mode 100644
index 0000000..8a686e5
--- /dev/null
+++ b/modules/catalog/tech-notes/apply_update_flow.png
Binary files differ
