| <div role="region"><a href="#" class="skipToContent_ZgBM">Skip to main content</a></div><nav class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Navigation bar toggle" class="navbar__toggle clean-btn" type="button" tabindex="0"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><div class="navbar__logo"><img src="/img/icons/128.png" alt="Apache Submarine Site Logo" class="themedImage_W2Cr themedImage--light_TfLj"><img src="/img/icons/128.png" alt="Apache Submarine Site Logo" class="themedImage_W2Cr themedImage--dark_oUvU"></div><b class="navbar__title">Apache Submarine</b></a><a class="navbar__item navbar__link navbar__link--active" href="/docs/gettingStarted/quickstart">Docs</a><a class="navbar__item navbar__link" href="/docs/api/environment">API</a><a class="navbar__item navbar__link" href="/docs/download">Download</a></div><div class="navbar__items navbar__items--right"><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link"><span><svg viewBox="0 0 24 24" width="20" height="20" aria-hidden="true" class="iconLanguage_dNtB"><path fill="currentColor" d="M12.87 15.07l-2.54-2.51.03-.03c1.74-1.94 2.98-4.17 3.71-6.53H17V4h-7V2H8v2H1v1.99h11.17C11.5 7.92 10.44 9.75 9 11.35 8.07 10.32 7.3 9.19 6.69 8h-2c.73 1.63 1.73 3.17 2.98 4.56l-5.09 5.02L4 19l5-5 3.11 3.11.76-2.04zM18.5 10h-2L12 22h2l1.12-3h4.75L21 22h2l-4.5-12zm-2.62 7l1.62-4.33L19.12 17h-3.24z"></path></svg><span>English</span></span></a><ul class="dropdown__menu"><li><a href="/docs/designDocs/storage-implementation" target="_self" rel="noopener noreferrer" class="dropdown__link dropdown__link--active">English</a></li><li><a href="/zh-cn/docs/designDocs/storage-implementation" target="_self" rel="noopener noreferrer" class="dropdown__link">δΈζ</a></li></ul></div><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a class="navbar__link" aria-haspopup="true" aria-expanded="false" role="button" href="/docs/gettingStarted/quickstart">0.8.0</a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/docs/next/designDocs/storage-implementation">master π</a></li><li><a aria-current="page" class="dropdown__link dropdown__link--active" href="/docs/designDocs/storage-implementation">0.8.0</a></li><li><a class="dropdown__link" href="/docs/0.7.0/designDocs/storage-implementation">0.7.0</a></li><li><a class="dropdown__link" href="/docs/0.6.0/designDocs/storage-implementation">0.6.0</a></li><li><a class="dropdown__link" href="/versions">All versions</a></li></ul></div><a href="https://github.com/apache/submarine" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link">GitHub<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_I5OW"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link">Apache</a><ul class="dropdown__menu"><li><a href="https://www.apache.org/foundation/how-it-works.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Apache Software Foundation<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_I5OW"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li><a href="https://www.apache.org/events/current-event" target="_blank" rel="noopener noreferrer" class="dropdown__link">Events<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_I5OW"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li><a href="https://www.apache.org/licenses/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Apache License<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_I5OW"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Thanks<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_I5OW"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li><a href="https://www.apache.org/security/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Security<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_I5OW"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Sponsorship<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_I5OW"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li></ul></div><div class="toggle_S7eR colorModeToggle_vKtC"><button class="clean-btn toggleButton_rCf9 toggleButtonDisabled_Pu9x" type="button" disabled="" title="Switch between dark and light mode (currently light mode)" aria-label="Switch between dark and light mode (currently light mode)"><svg viewBox="0 0 24 24" width="24" height="24" class="lightToggleIcon_v35p"><path fill="currentColor" d="M12,9c1.65,0,3,1.35,3,3s-1.35,3-3,3s-3-1.35-3-3S10.35,9,12,9 M12,7c-2.76,0-5,2.24-5,5s2.24,5,5,5s5-2.24,5-5 S14.76,7,12,7L12,7z M2,13l2,0c0.55,0,1-0.45,1-1s-0.45-1-1-1l-2,0c-0.55,0-1,0.45-1,1S1.45,13,2,13z M20,13l2,0c0.55,0,1-0.45,1-1 s-0.45-1-1-1l-2,0c-0.55,0-1,0.45-1,1S19.45,13,20,13z M11,2v2c0,0.55,0.45,1,1,1s1-0.45,1-1V2c0-0.55-0.45-1-1-1S11,1.45,11,2z M11,20v2c0,0.55,0.45,1,1,1s1-0.45,1-1v-2c0-0.55-0.45-1-1-1C11.45,19,11,19.45,11,20z M5.99,4.58c-0.39-0.39-1.03-0.39-1.41,0 c-0.39,0.39-0.39,1.03,0,1.41l1.06,1.06c0.39,0.39,1.03,0.39,1.41,0s0.39-1.03,0-1.41L5.99,4.58z M18.36,16.95 c-0.39-0.39-1.03-0.39-1.41,0c-0.39,0.39-0.39,1.03,0,1.41l1.06,1.06c0.39,0.39,1.03,0.39,1.41,0c0.39-0.39,0.39-1.03,0-1.41 L18.36,16.95z M19.42,5.99c0.39-0.39,0.39-1.03,0-1.41c-0.39-0.39-1.03-0.39-1.41,0l-1.06,1.06c-0.39,0.39-0.39,1.03,0,1.41 s1.03,0.39,1.41,0L19.42,5.99z M7.05,18.36c0.39-0.39,0.39-1.03,0-1.41c-0.39-0.39-1.03-0.39-1.41,0l-1.06,1.06 c-0.39,0.39-0.39,1.03,0,1.41s1.03,0.39,1.41,0L7.05,18.36z"></path></svg><svg viewBox="0 0 24 24" width="24" height="24" class="darkToggleIcon_nQuB"><path fill="currentColor" d="M9.37,5.51C9.19,6.15,9.1,6.82,9.1,7.5c0,4.08,3.32,7.4,7.4,7.4c0.68,0,1.35-0.09,1.99-0.27C17.45,17.19,14.93,19,12,19 c-3.86,0-7-3.14-7-7C5,9.07,6.81,6.55,9.37,5.51z M12,3c-4.97,0-9,4.03-9,9s4.03,9,9,9s9-4.03,9-9c0-0.46-0.04-0.92-0.1-1.36 c-0.98,1.37-2.58,2.26-4.4,2.26c-2.98,0-5.4-2.42-5.4-5.4c0-1.81,0.89-3.42,2.26-4.4C12.92,3.04,12.46,3,12,3L12,3z"></path></svg></button></div><div class="navbar__search"><span aria-label="expand searchbar" role="button" class="search-icon" tabindex="0"></span><input type="search" id="search_input_react" placeholder="Search" aria-label="Search" class="navbar__search-input search-bar"></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div class="main-wrapper"><div class="docPage_P2Lg"><button aria-label="Scroll back to top" class="clean-btn theme-back-to-top-button backToTopButton_RiI4" type="button"></button><aside class="theme-doc-sidebar-container docSidebarContainer_rKC_"><div class="sidebar_RiAD"><nav class="menu thin-scrollbar menu_izAj"><ul class="theme-doc-sidebar-menu menu__list"><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/gettingStarted/quickstart">Getting Started</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/userDocs/submarine-sdk/submarine-cli">User Docs</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/devDocs/">Developer Docs</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/community/">Community</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret menu__link--active" aria-expanded="true" href="/docs/designDocs/architecture-and-requirements">Design Docs</a></div><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/designDocs/architecture-and-requirements">Architecture and Requirment</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/designDocs/implementation-notes">Implementation Notes</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/designDocs/environments-implementation">Environments Implementation</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/designDocs/experiment-implementation">Experiment Implementation</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/designDocs/notebook-implementation">Notebook Implementation</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link menu__link--active" aria-current="page" tabindex="0" href="/docs/designDocs/storage-implementation">Storage Implementation</a></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-2 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" tabindex="0" href="/docs/designDocs/submarine-server/architecture">Submarine Server</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-2 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" tabindex="0" href="/docs/designDocs/wip-designs/submarine-launcher">WIP Design Docs</a></div></li></ul></li></ul></nav></div></aside><main class="docMainContainer_TCnq"><div class="container padding-top--md padding-bottom--lg"><div class="row"><div class="col docItemCol_DM6M"><div class="docItemContainer_vinB"><article><nav class="theme-doc-breadcrumbs breadcrumbsContainer_Xlws" aria-label="breadcrumbs"><ul class="breadcrumbs" itemscope="" itemtype="https://schema.org/BreadcrumbList"><li class="breadcrumbs__item"><a class="breadcrumbs__link" href="/">π </a></li><li itemscope="" itemprop="itemListElement" itemtype="https://schema.org/ListItem" class="breadcrumbs__item"><span class="breadcrumbs__link" itemprop="item name">Design Docs</span><meta itemprop="position" content="1"></li><li itemscope="" itemprop="itemListElement" itemtype="https://schema.org/ListItem" class="breadcrumbs__item breadcrumbs__item--active"><span class="breadcrumbs__link" itemprop="item name">Storage Implementation</span><meta itemprop="position" content="2"></li></ul></nav><span class="theme-doc-version-badge badge badge--secondary">Version: 0.8.0</span><div class="tocCollapsible_jdIR theme-doc-toc-mobile tocMobile_TmEX"><button type="button" class="clean-btn tocCollapsibleButton_Fzxq">On this page</button></div><div class="theme-doc-markdown markdown"><header><h1>Storage Implementation</h1></header><h2 class="anchor anchorWithStickyNavbar_mojV" id="ml-related-objects-and-their-storages">ML-related objects and their storages<a class="hash-link" href="#ml-related-objects-and-their-storages" title="Direct link to heading">β</a></h2><p>First let's look at what user will interact for most of the time: </p><ul><li>Notebook </li><li>Experiment</li><li>Model Servings</li></ul><div class="codeBlockContainer_I0IT theme-code-block"><div class="codeBlockContent_wNvx" style="color:#bfc7d5;background-color:#292d3e"><pre tabindex="0" class="prism-code language-text codeBlock_jd64 thin-scrollbar"><code class="codeBlockLines_mRuA"><span class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> +---------+ +------------+</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> |Logs |<--+|Notebook |</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> +----------+ +---------+ +------------+ +----------------+</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> |Trackings | <-+|Experiment |<--+>|Model Artifacts |</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> +----------+ +-----------------+ +------------+ +----------------+</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> +----------+<---+|ML-related Metric|<--+Servings |</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> |tf.events | +-----------------+ +------------+</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> +----------+ ^ +-----------------+</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> + | Environments |</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> +----------------------+ | |</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> +-----------------+ | Submarine Metastore | | Dependencies |</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> |Code | +----------------------+ | |</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> +-----------------+ |Experiment Meta | | Docker Images |</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> +----------------------+ +-----------------+</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> |Model Store Meta |</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> +----------------------+</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> |Model Serving Meta |</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> +----------------------+</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> |Notebook meta |</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> +----------------------+</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> |Experiment Templates |</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> +----------------------+</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> |Environments Meta |</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> +----------------------+</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" title="Copy" class="copyButton_eDfN clean-btn"><span class="copyButtonIcons_W9eQ" aria-hidden="true"><svg class="copyButtonIcon_XEyF" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_i9w9" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div><p>First of all, all the notebook-sessions / experiments / model-serving instances) are more or less interact with following storage objects:</p><ul><li>Logs for these tasks for troubleshooting. </li><li>ML-related metrics such as loss, epoch, etc. (in contrast of system metrics such as CPU/memory usage, etc.)<ul><li>There're different types of ML-related metrics, for Tensorflow/pytorch, they can use tf.events and get visualizations on tensorboard. </li><li>Or they can use tracking APIs (such as Submarine tracking, mlflow tracking, etc.) to output customized tracking results for non TF/Pytorch workloads. </li></ul></li><li>Training jobs of experiment typically generate model artifacts (files) which need persisted, and both of notebook, model serving needs to load model artifacts from persistent storage. </li><li>There're various of meta information, such as experiment meta, model registry, model serving, notebook, experiment, environment, etc. We need be able to read these meta information back.</li><li>We also have code for experiment (like training/batch-prediction), notebook (ipynb), and model servings.</li><li>And notebook/experiments/model-serving need depend on environments (dependencies such as pip, and Docker Images).</li></ul><h3 class="anchor anchorWithStickyNavbar_mojV" id="implementation-considerations-for-ml-related-objects">Implementation considerations for ML-related objects<a class="hash-link" href="#implementation-considerations-for-ml-related-objects" title="Direct link to heading">β</a></h3><table><thead><tr><th>Object Type</th><th>Characteristics</th><th>Where to store</th></tr></thead><tbody><tr><td>Metrics: tf.events</td><td>Time series data with k/v, appendable to file</td><td>Local/EBS, HDFS, Cloud Blob Storage</td></tr><tr><td>Metrics: other tracking metrics</td><td>Time series data with k/v, appendable to file</td><td>Local, HDFS, Cloud Blob Storage, Database</td></tr><tr><td>Logs</td><td>Large volumes, #files are potentially huge.</td><td>Local (temporary), HDFS (need aggregation), Cloud Blob Storage</td></tr><tr><td>Submarine Metastore</td><td>CRUD operations for small meta data.</td><td>Database</td></tr><tr><td>Model Artifacts</td><td>Size varies for model (from KBs to GBs). #files are potentially huge.</td><td>HDFS, Cloud Blob Storage</td></tr><tr><td>Code</td><td>Need version control. (Please find detailed discussions below for code storage and localization)</td><td>Tarball on HDFS/Cloud Blog Storage, or Git</td></tr><tr><td>Environment (Dependencies, Docker Image)</td><td></td><td>Public/private environment repo (like Conda channel), Docker registry.</td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_mojV" id="detailed-discussions">Detailed discussions<a class="hash-link" href="#detailed-discussions" title="Direct link to heading">β</a></h3><h4 class="anchor anchorWithStickyNavbar_mojV" id="store-code-for-experimentnotebookmodel-serving">Store code for experiment/notebook/model-serving<a class="hash-link" href="#store-code-for-experimentnotebookmodel-serving" title="Direct link to heading">β</a></h4><p>There're following ways to get experiment code: </p><p><strong>1) Code is part of Git repo:</strong> (<strong><em><u>Recommended</u></em></strong>)</p><p>This is our recommended approach, once code is part of Git, it will be stored in version control, any change will be tracked, and much easier for users to trace back what change triggered a new bug, etc.</p><p><strong>2) Code is part of Docker image:</strong> </p><p><strong><em>This is an anti-pattern and we will NOT recommend you to use it</em></strong>, Docker image can be used to include ANYTHING, like dependencies, the code you will execute, or even data. But this doesn't mean you should do it. We recommend to use Docker image ONLY for libraries/dependencies.</p><p>Making code to be part of Docker image makes hard to edit code (if you want to update a value in your Python file, you will have to recreate the Docker image, push it and rerun it).</p><p><strong>3) Code is part of S3/HDFS/ABFS:</strong> </p><p>User may want to store their training code to a tarball on a shared storage. Submarine need to download code from remote storage to the launched container before running the code. </p><h4 class="anchor anchorWithStickyNavbar_mojV" id="localization-of-experimentnotebookmodel-serving-code">Localization of experiment/notebook/model-serving code<a class="hash-link" href="#localization-of-experimentnotebookmodel-serving-code" title="Direct link to heading">β</a></h4><p>To make user experiences keeps same across different environment, we will localize code to a same folder after the container is launched, preferably <code>/code</code></p><p>For example, there's a git repo need to be synced up for an experiment/notebook/model-serving (example above):</p><div class="codeBlockContainer_I0IT theme-code-block"><div class="codeBlockContent_wNvx" style="color:#bfc7d5;background-color:#292d3e"><pre tabindex="0" class="prism-code language-text codeBlock_jd64 thin-scrollbar"><code class="codeBlockLines_mRuA"><span class="token-line" style="color:#bfc7d5"><span class="token plain">experiment: #Or notebook, model-serving</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> name: "abc",</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> environment: "team-default-ml-env"</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> ... (other fields)</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> code:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> sync_mode: git</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> url: "https://foo.com/training-job.git" </span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" title="Copy" class="copyButton_eDfN clean-btn"><span class="copyButtonIcons_W9eQ" aria-hidden="true"><svg class="copyButtonIcon_XEyF" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_i9w9" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div><p>After localize, <code>training-job/</code> will be placed under <code>/code</code> </p><p>When we running on K8s environment, we can use K8s's initContainer and emptyDir to do these things for us. K8s POD spec (generated by Submarine server instead of user, user should NEVER edit K8s spec, that's too unfriendly to data-scientists): </p><div class="codeBlockContainer_I0IT theme-code-block"><div class="codeBlockContent_wNvx" style="color:#bfc7d5;background-color:#292d3e"><pre tabindex="0" class="prism-code language-text codeBlock_jd64 thin-scrollbar"><code class="codeBlockLines_mRuA"><span class="token-line" style="color:#bfc7d5"><span class="token plain">apiVersion: v1</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">kind: Pod</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">metadata:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> name: experiment-abc</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">spec:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> containers:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> - name: experiment-task</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> image: training-job</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> volumeMounts:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> - name: code-dir</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> mountPath: /code</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> initContainers:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> - name: git-localize</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> image: git-sync</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> command: "git clone .. /code/"</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> volumeMounts:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> - name: code-dir</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> mountPath: /code</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> volumes:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> - name: code-dir</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> emptyDir: {}</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" title="Copy" class="copyButton_eDfN clean-btn"><span class="copyButtonIcons_W9eQ" aria-hidden="true"><svg class="copyButtonIcon_XEyF" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_i9w9" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div><p>The above K8s spec create a code-dir and mount it to <code>/code</code> to launched containers. The initContainer <code>git-localize</code> uses <code>https://github.com/kubernetes/git-sync</code> to do the sync up. (If other storages are used such as s3, we can use similar initContainer approach to download contents)</p><h2 class="anchor anchorWithStickyNavbar_mojV" id="system-related-metricslogs-and-their-storages">System-related metrics/logs and their storages<a class="hash-link" href="#system-related-metricslogs-and-their-storages" title="Direct link to heading">β</a></h2><p>Other than ML-related objects, we have system-related objects, including: </p><ul><li>Daemon logs (like logs of Submarine server). </li><li>Logs for other dependency components (like Kubernetes logs when running on K8s). </li><li>System metrics (Physical resource usages by daemons, launched training containers, etc.). </li></ul><p>All these information should be handled by 3rd party system, such as Grafana, Prometheus, etc. And system admins are responsible to setup these infrastructures, dashboard. Users of submarine should NOT interact with system related metrics/logs. It is system admin's responsibility.</p><h2 class="anchor anchorWithStickyNavbar_mojV" id="attachable-volumes">Attachable Volumes<a class="hash-link" href="#attachable-volumes" title="Direct link to heading">β</a></h2><p>It is possible user has needs to have an attachable volume for their experiment / notebook, this is especially useful for notebook storage, since contents of notebook can be automatically saved, and it can be used as user's home folder. </p><p>Downside of attachable volume is, it is not versioned, even notebook is mainly used for adhoc exploring tasks, an unversioned notebook file can lead to maintenance issues in the future. </p><p>Since this is a common requirement, we can consider to support attachable volumes in Submarine in a long run, but with relatively lower priority.</p><h2 class="anchor anchorWithStickyNavbar_mojV" id="in-scope--out-of-scope">In-scope / Out-of-scope<a class="hash-link" href="#in-scope--out-of-scope" title="Direct link to heading">β</a></h2><p> Describe what Submarine project should own and what Submarine project should NOT own.</p></div><footer class="theme-doc-footer docusaurus-mt-lg"><div class="theme-doc-footer-edit-meta-row row"><div class="col"><a href="https://github.com/apache/submarine/edit/master/website/versioned_docs/version-0.8.0/designDocs/storage-implementation.md" target="_blank" rel="noreferrer noopener" class="theme-edit-this-page"><svg fill="currentColor" height="20" width="20" viewBox="0 0 40 40" class="iconEdit_dcUD" aria-hidden="true"><g><path d="m34.5 11.7l-3 3.1-6.3-6.3 3.1-3q0.5-0.5 1.2-0.5t1.1 0.5l3.9 3.9q0.5 0.4 0.5 1.1t-0.5 1.2z m-29.5 17.1l18.4-18.5 6.3 6.3-18.4 18.4h-6.3v-6.2z"></path></g></svg>Edit this page</a></div><div class="col lastUpdated_foO9"></div></div></footer></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Docs pages navigation"><div class="pagination-nav__item"><a class="pagination-nav__link" href="/docs/designDocs/notebook-implementation"><div class="pagination-nav__sublabel">Previous</div><div class="pagination-nav__label">Notebook Implementation</div></a></div><div class="pagination-nav__item pagination-nav__item--next"><a class="pagination-nav__link" href="/docs/designDocs/submarine-server/architecture"><div class="pagination-nav__sublabel">Next</div><div class="pagination-nav__label">Submarine Server Implementation</div></a></div></nav></div></div><div class="col col--3"><div class="tableOfContents_cNA8 thin-scrollbar theme-doc-toc-desktop"><ul class="table-of-contents table-of-contents__left-border"><li><a href="#ml-related-objects-and-their-storages" class="table-of-contents__link toc-highlight">ML-related objects and their storages</a><ul><li><a href="#implementation-considerations-for-ml-related-objects" class="table-of-contents__link toc-highlight">Implementation considerations for ML-related objects</a></li><li><a href="#detailed-discussions" class="table-of-contents__link toc-highlight">Detailed discussions</a></li></ul></li><li><a href="#system-related-metricslogs-and-their-storages" class="table-of-contents__link toc-highlight">System-related metrics/logs and their storages</a></li><li><a href="#attachable-volumes" class="table-of-contents__link toc-highlight">Attachable Volumes</a></li><li><a href="#in-scope--out-of-scope" class="table-of-contents__link toc-highlight">In-scope / Out-of-scope</a></li></ul></div></div></div></div></main></div></div><footer class="footer footer--dark"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">Docs</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/docs/gettingStarted/quickstart">Getting Started</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/api/environment">API docs</a></li></ul></div><div class="col footer__col"><div class="footer__title">Community</div><ul class="footer__items"><li class="footer__item"><a href="https://stackoverflow.com/questions/tagged/apache-submarine" target="_blank" rel="noopener noreferrer" class="footer__link-item">Stack Overflow<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_I5OW"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li class="footer__item"><a href="https://s.apache.org/slack-invite" target="_blank" rel="noopener noreferrer" class="footer__link-item">Slack<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_I5OW"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li></ul></div><div class="col footer__col"><div class="footer__title">More</div><ul class="footer__items"><li class="footer__item"><a href="https://medium.com/@apache.submarine" target="_blank" rel="noopener noreferrer" class="footer__link-item">Blog</a></li><li class="footer__item"><a href="https://github.com/apache/submarine" target="_blank" rel="noopener noreferrer" class="footer__link-item">GitHub<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_I5OW"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li></ul></div></div><div class="footer__bottom text--center"><div class="margin-bottom--sm"><a href="https://www.apache.org/" target="_blank" rel="noopener noreferrer" class="footerLogoLink_gHmE"><img src="https://hadoop.apache.org/asf_logo_wide.png" alt="Apache Open Source Logo" class="themedImage_W2Cr themedImage--light_TfLj footer__logo"><img src="https://hadoop.apache.org/asf_logo_wide.png" alt="Apache Open Source Logo" class="themedImage_W2Cr themedImage--dark_oUvU footer__logo"></a></div><div class="footer__copyright">Apache Submarine, Submarine, Apache, the Apache feather logo, and the Apache Submarine project logo are |
