| <!-- |
| Documentation/_templates/layout.html |
| |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. The |
| ASF licenses this file to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance with the |
| License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, WITHOUT |
| WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the |
| License for the specific language governing permissions and limitations |
| under the License. |
| --> |
| |
| |
| |
| <!DOCTYPE html> |
| <html class="writer-html5" lang="en"> |
| <head> |
| <meta charset="utf-8" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" /> |
| |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <title>Porting to the BCM2711 (Raspberry Pi 4B) — NuttX latest documentation</title> |
| <link rel="stylesheet" type="text/css" href="../../_static/pygments.css" /> |
| <link rel="stylesheet" type="text/css" href="../../_static/css/theme.css" /> |
| <link rel="stylesheet" type="text/css" href="../../_static/copybutton.css" /> |
| <link rel="stylesheet" type="text/css" href="../../_static/design-style.1e8bd061cd6da7fc9cf755528e8ffc24.min.css" /> |
| <link rel="stylesheet" type="text/css" href="../../_static/custom.css" /> |
| |
| |
| <link rel="shortcut icon" href="../../_static/favicon.ico"/> |
| <script src="../../_static/jquery.js"></script> |
| <script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script> |
| <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script> |
| <script src="../../_static/doctools.js"></script> |
| <script src="../../_static/sphinx_highlight.js"></script> |
| <script src="../../_static/clipboard.min.js"></script> |
| <script src="../../_static/copybutton.js"></script> |
| <script src="../../_static/design-tabs.js"></script> |
| <script src="../../_static/js/theme.js"></script> |
| <link rel="index" title="Index" href="../../genindex.html" /> |
| <link rel="search" title="Search" href="../../search.html" /> |
| <link rel="next" title="The case of ARM CM4 & cxd32xx @NuttX12.4.0" href="port_arm_cm4.html" /> |
| <link rel="prev" title="The list of related kernel configurations" href="../port_relatedkernelconfigrations.html" /> |
| </head> |
| |
| <body class="wy-body-for-nav"> |
| <div class="wy-grid-for-nav"> |
| <nav data-toggle="wy-nav-shift" class="wy-nav-side"> |
| <div class="wy-side-scroll"> |
| <div class="wy-side-nav-search" > |
| |
| <a href="../../index.html" class="icon icon-home"> NuttX |
| |
| |
| |
| </a> |
| |
| <!-- this version selector is quite ugly, should be probably replaced by something |
| more modern --> |
| |
| <div class="version-selector"> |
| <select onchange="javascript:location.href = this.value;"> |
| |
| <option value="../../../latest" selected="selected">latest</option> |
| |
| <option value="../../../10.0.0" >10.0.0</option> |
| |
| <option value="../../../10.0.1" >10.0.1</option> |
| |
| <option value="../../../10.1.0" >10.1.0</option> |
| |
| <option value="../../../10.2.0" >10.2.0</option> |
| |
| <option value="../../../10.3.0" >10.3.0</option> |
| |
| <option value="../../../11.0.0" >11.0.0</option> |
| |
| <option value="../../../12.0.0" >12.0.0</option> |
| |
| <option value="../../../12.1.0" >12.1.0</option> |
| |
| <option value="../../../12.2.0" >12.2.0</option> |
| |
| <option value="../../../12.2.1" >12.2.1</option> |
| |
| <option value="../../../12.3.0" >12.3.0</option> |
| |
| <option value="../../../12.4.0" >12.4.0</option> |
| |
| <option value="../../../12.5.0" >12.5.0</option> |
| |
| <option value="../../../12.5.1" >12.5.1</option> |
| |
| <option value="../../../12.6.0" >12.6.0</option> |
| |
| <option value="../../../12.7.0" >12.7.0</option> |
| |
| <option value="../../../12.8.0" >12.8.0</option> |
| |
| <option value="../../../12.9.0" >12.9.0</option> |
| |
| <option value="../../../12.10.0" >12.10.0</option> |
| |
| <option value="../../../12.11.0" >12.11.0</option> |
| |
| </select> |
| </div> |
| |
| |
| <div role="search"> |
| <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get"> |
| <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" /> |
| <input type="hidden" name="check_keywords" value="yes" /> |
| <input type="hidden" name="area" value="default" /> |
| </form> |
| </div> |
| |
| </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu"> |
| <p class="caption" role="heading"><span class="caption-text">Table of Contents</span></p> |
| <ul class="current"> |
| <li class="toctree-l1"><a class="reference internal" href="../../index.html">Home</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../introduction/index.html">Introduction</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../quickstart/index.html">Getting Started</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../contributing/index.html">Contributing</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../introduction/inviolables.html">The Inviolable Principles of NuttX</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../platforms/index.html">Supported Platforms</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../components/index.html">OS Components</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../applications/index.html">Applications</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../implementation/index.html">Implementation Details</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../reference/index.html">API Reference</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../faq/index.html">FAQ</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../debugging/index.html">Debugging</a></li> |
| <li class="toctree-l1 current"><a class="reference internal" href="../index.html">Guides</a><ul class="current"> |
| <li class="toctree-l2"><a class="reference internal" href="../nfs.html">NFS Client How-To</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../usbtrace.html">USB Device Trace</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../simulator.html">Simulator</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../rndis.html">How to use RNDIS</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../drivers.html">Drivers</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../cpp_cmake.html">C++ Example using CMake</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../pysimcoder.html">pysimCoder integration with NuttX</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../customboards.html">Custom Boards How-To</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../customapps.html">Custom Apps How-to</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../citests.html">Running CI Test Locally</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../zerolatencyinterrupts.html">High Performance: Zero Latency Interrupts, Maskable Nested Interrupts</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../fortify.html">Fortify</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../nestedinterrupts.html">Nested Interrupts</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../ofloader.html">Open Flash Loader</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../testingtcpip.html">Testing TCP/IP Network Stacks</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../automounter.html">Auto-Mounter</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../stm32nullpointer.html">STM32 Null Pointer Detection</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../stm32ccm.html">STM32 CCM Allocator</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../etcromfs.html">etc romfs</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../thread_local_storage.html">Thread Local Storage</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../devicetree.html">Device Tree</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../changing_systemclockconfig.html">Changing the System Clock Configuration</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../usingkernelthreads.html">Using Kernel Threads</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../armv7m_runtimestackcheck.html">ARMv7-M Run Time Stack Checking</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../include_files_board_h.html">Including Files in board.h</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../specialstuff_in_nuttxheaderfiles.html">Why can’t I put my special stuff in NuttX header files?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../kernel_threads_with_custom_stacks.html">Kernel Threads with Custom Stacks</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../versioning_and_task_names.html">Versioning and Task Names</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../logging_rambuffer.html">Logging to a RAM Buffer</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../ipv6.html">IPv6</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../integrate_newlib.html">Integrating with Newlib</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../protected_build.html">NuttX Protected Build</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../platform_directories.html">Platform Directories</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../port_drivers_to_stm32f7.html">Porting Drivers to the STM32 F7</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../semihosting.html">Semihosting</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../renode.html">Run NuttX on Renode</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../signal_events_interrupt_handlers.html">Signaling Events from Interrupt Handlers</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../signaling_sem_priority_inheritance.html">Signaling Semaphores and Priority Inheritance</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../smaller_vector_tables.html">Smaller Vector Tables</a></li> |
| <li class="toctree-l2 current"><a class="reference internal" href="../port.html">How to port</a><ul class="current"> |
| <li class="toctree-l3"><a class="reference internal" href="../port_bootsequence.html">The diagram of boot sequence</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../port_relatedkernelconfigrations.html">The list of related kernel configurations</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../port.html#porting-procedure">Porting procedure</a></li> |
| <li class="toctree-l3 current"><a class="reference internal" href="../port.html#porting-case-studies">Porting Case Studies</a><ul class="current"> |
| <li class="toctree-l4 current"><a class="current reference internal" href="#">Porting to the BCM2711 (Raspberry Pi 4B)</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="#researching">Researching</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="#adding-to-the-source-tree">Adding to the source tree</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="#mapping-out-the-chip">Mapping out the chip</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="#figuring-out-the-boot">Figuring out the boot</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l4"><a class="reference internal" href="port_arm_cm4.html">The case of ARM CM4 & cxd32xx @NuttX12.4.0</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../updating_release_system_elf.html">Updating a Release System with ELF Programs</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../partially_linked_elf.html">ELF Programs – With Symbol Tables</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../fully_linked_elf.html">ELF Programs – No Symbol Tables</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../building_nuttx_with_app_out_of_src_tree.html">Building NuttX with Applications Outside the Source Tree</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../building_uclibcpp.html">Building uClibc++</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../custom_app_directories.html">Custom Application Directories</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../multiple_nsh_sessions.html">Multiple NSH Sessions</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../nsh_network_link_management.html">NSH Network Link Management</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../ram_rom_disks.html">RAM Disks and ROM Disks</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../reading_can_msgs.html">Reading CAN Messages</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../remove_device_drivers_nsh.html">Removing Device Drivers with NSH</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../rust.html">Rust in NuttX</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../optee.html">Interfacing with OP-TEE</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l1"><a class="reference internal" href="../../glossary.html">Glossary</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../logos/index.html">NuttX Logos</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../_tags/tagsindex.html">Tags</a></li> |
| </ul> |
| |
| </div> |
| </div> |
| </nav> |
| |
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" > |
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> |
| <a href="../../index.html">NuttX</a> |
| </nav> |
| |
| <div class="wy-nav-content"> |
| <div class="rst-content"> |
| <div role="navigation" aria-label="Page navigation"> |
| <ul class="wy-breadcrumbs"> |
| <li><a href="../../index.html" class="icon icon-home" aria-label="Home"></a></li> |
| <li class="breadcrumb-item"><a href="../index.html">Guides</a></li> |
| <li class="breadcrumb-item"><a href="../port.html">How to port</a></li> |
| <li class="breadcrumb-item active">Porting to the BCM2711 (Raspberry Pi 4B)</li> |
| <li class="wy-breadcrumbs-aside"> |
| <a href="https://github.com/apache/nuttx/blob/master/Documentation/guides/porting-case-studies/bcm2711-rpi4b.rst" class="fa fa-github"> Edit on GitHub</a> |
| </li> |
| </ul> |
| <hr/> |
| </div> |
| <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> |
| <div itemprop="articleBody"> |
| |
| <section id="porting-to-the-bcm2711-raspberry-pi-4b"> |
| <h1>Porting to the BCM2711 (Raspberry Pi 4B)<a class="headerlink" href="#porting-to-the-bcm2711-raspberry-pi-4b" title="Permalink to this heading"></a></h1> |
| <p>This port was completed for the 12.7.0 version of the NuttX kernel, and was contributed by Matteo Golin.</p> |
| <p>The pull request with this initial support can be found at <a class="reference external" href="https://github.com/apache/nuttx/pull/15188">apache/nuttx/pull/15188</a>.</p> |
| <p>The port required support to be written for a new chip (the BCM2711) and a new board. Matteo created journal entries |
| while working on the initial port, which can be found <a class="reference external" href="https://linguini1.github.io/blog/2024/12/25/nuttx-bcm2711.html">on his blog</a>. The details below are a more concise summary of the |
| porting process.</p> |
| <section id="researching"> |
| <h2>Researching<a class="headerlink" href="#researching" title="Permalink to this heading"></a></h2> |
| <p>The first step to porting a board to NuttX was researching the board and how NuttX works.</p> |
| <p>The BCM2711 is a quad-core ARM Cortex A72 based SoC, and it supports both aarch64 and 32 bit ARM architectures. I |
| focused on the aarch64 implementation only in this port. My first step was determining other boards already in the NuttX |
| kernel that used the aarch64 architecture, because that gives me a starting point to porting this new chip and board.</p> |
| <p>I primarily used the blog posts written by Lup Yuen Lee about porting NuttX to the PinePhone, another ARM Cortex-A based |
| device. The articles are listed <a class="reference external" href="https://github.com/lupyuen/pinephone-nuttx">here</a>. Lup’s articles provided me with an |
| understanding of the NuttX boot process, as well as which files from the aarch64 support on NuttX were pulled into the |
| build process for booting. He also showed how he created an initial UART driver using the NuttX structure for UART |
| drivers, which allowed him to get NSH appearing in the console.</p> |
| <p>Finally, I also of course needed the BCM2711 datasheet in order to figure out which registers were available to me for |
| creating peripheral drivers. The BCM2711 datasheet isn’t exceptionally detailed on many of the features on the SoC, but |
| it did provide enough detail to set up interrupts and get UART working.</p> |
| </section> |
| <section id="adding-to-the-source-tree"> |
| <h2>Adding to the source tree<a class="headerlink" href="#adding-to-the-source-tree" title="Permalink to this heading"></a></h2> |
| <p>In order to build my code with the NuttX build system, I would have to add the board and the BCM2711 chip to the source |
| tree for NuttX. This way, it would appear as an available configuration via the <code class="docutils literal notranslate"><span class="pre">tools/configure.sh</span></code> script and I |
| could select options for it with <code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">menuconfig</span></code>.</p> |
| <p>The first thing to do was to add the chip, which goes under the <code class="docutils literal notranslate"><span class="pre">arch/arm64</span></code> directory because it is an ARM 64 bit |
| SoC. The chip directory must be added in two places: <code class="docutils literal notranslate"><span class="pre">arch/arm64/include/bcm2711</span></code> and <code class="docutils literal notranslate"><span class="pre">arch/arm64/src/bcm2711</span></code>. C |
| files go in the <code class="docutils literal notranslate"><span class="pre">src</span></code> directory with some header files, and some specific header files go in the <code class="docutils literal notranslate"><span class="pre">include</span></code> |
| directory.</p> |
| <p>In addition, in order to make the BCM2711 visible as a supported chip, I had to add it as an option in |
| <code class="docutils literal notranslate"><span class="pre">arch/arm64/Kconfig</span></code>. In order to do this, I just copy-pasted the entry for the Allwinner A64, since the two chips |
| were very similar. I had to change a few fields (for instance, selecting <code class="docutils literal notranslate"><span class="pre">ARCH_CORTEX_A72</span></code> instead of |
| <code class="docutils literal notranslate"><span class="pre">ARCH_CORTEX_A53</span></code>), but this was relatively simple to complete with the information about the SoC. I also needed to |
| specify <code class="docutils literal notranslate"><span class="pre">ARMV8A_HAVE_GICv2</span></code>, since that is the interrupt controller used by the BCM2711. <code class="docutils literal notranslate"><span class="pre">ARCH_HAVE_MULTICPU</span></code> |
| because it is a quad-core, and <code class="docutils literal notranslate"><span class="pre">ARCH_USE_MMU</span></code> because it has a memory management unit.</p> |
| <p>I also needed to now add the Raspberry Pi 4B board to the source tree. To do this, I copied the board folder for the |
| PinePhone (<code class="docutils literal notranslate"><span class="pre">boards/arm64/a64/pinephone</span></code>) and renamed it <code class="docutils literal notranslate"><span class="pre">raspberrypi-4b</span></code>. I also deleted many of the files in this |
| folder since they weren’t applicable to the Pi 4B, and substituted all mentions of the PinePhone with the Raspberry Pi |
| 4B (in path names and header include guards).</p> |
| <p>I then added the Pi 4B to the list of supported boards in <code class="docutils literal notranslate"><span class="pre">boards/Kconfig</span></code>. For this, I just needed to create an entry |
| with the name <code class="docutils literal notranslate"><span class="pre">ARCH_BOARD_RASPBERRYPI_4B</span></code> and write that it depends on the <code class="docutils literal notranslate"><span class="pre">ARCH_CHIP_BCM2711</span></code>. No additional |
| options necessary! In two other places in this file I also had to add some directives to make sure the Kconfig for the |
| board was found properly. These set <code class="docutils literal notranslate"><span class="pre">ARCH_BOARD</span></code> to the name of the board directory “raspberrypi-4b” when the Pi 4B was |
| selected, and <code class="docutils literal notranslate"><span class="pre">source</span></code>’d the Kconfig under <code class="docutils literal notranslate"><span class="pre">boards/arm64/bcm2711/raspberrypi-4b</span></code> when selected.</p> |
| <p>The default configuration for this board was copied from the PinePhone’s NSH configuration, which I modified to use the |
| correct board name, chip, and hardware specific settings. It was still incomplete because there was no code to actually |
| boot into NSH, but it was a starting point.</p> |
| <p>This was basically all I needed for the board to show up as a possible configuration in the source tree!</p> |
| </section> |
| <section id="mapping-out-the-chip"> |
| <h2>Mapping out the chip<a class="headerlink" href="#mapping-out-the-chip" title="Permalink to this heading"></a></h2> |
| <p>To start writing code for the BCM2711, I needed to map out the chip. This included the register addresses and the memory |
| mapping, which could all be found in the BCM2711 datasheet. From looking at other implementations, the register |
| addresses are usually defined as C macros and kept in header files under <code class="docutils literal notranslate"><span class="pre">arch/<architecture>/src/<chip>/hardware</span></code>. |
| This is where I put them as well, defining all the register mappings the different groups within individual files (i.e. |
| <code class="docutils literal notranslate"><span class="pre">bmc2711_i2c.h</span></code>, <code class="docutils literal notranslate"><span class="pre">bcm2711_spi.h</span></code>, etc.).</p> |
| <p>Many peripherals had groupings of memory-mapped registers, defined using a base address and then offsets from that |
| address to access the different fields. For instance, the two mini-SPI peripherals had the same structure, each with 12 |
| registers. The way I commonly saw these macros implemented was something like:</p> |
| <div class="highlight-c notranslate"><div class="highlight"><pre><span></span><span class="cp">#define BCM_AUX_SPI1_BASEADDR (BCM_AUX_BASEADDR + BCM_AUX_SPI1_OFFSET)</span> |
| |
| <span class="cp">#define BCM_AUX_SPI_CNTL0_REG_OFFSET (0x00) </span><span class="cm">/* SPI control register 0 */</span> |
| <span class="cm">/* ... more register offsets */</span> |
| |
| <span class="cm">/* This allows you to choose which SPI interface base address to get the register for. */</span> |
| |
| <span class="cp">#define BCM_AUX_SPI_CNTL0(base) ((base) + BCM_AUX_SPI_CNTL0_REG_OFFSET)</span> |
| </pre></div> |
| </div> |
| <p>In addition to the registers themselves, I also included macros to mask certain fields within the registers or set |
| certain values. This makes the code less error prone later, because any mistakes made while copying the long list of |
| fields and registers from the datasheet can be changed in one place.</p> |
| <div class="highlight-c notranslate"><div class="highlight"><pre><span></span><span class="cp">#define BCM_SPI_CNTL0_EN (1 << 11) </span><span class="cm">/* Enable SPI interface */</span> |
| </pre></div> |
| </div> |
| <p>In addition to the registers, I also had to map the interrupts. This was done in <code class="docutils literal notranslate"><span class="pre">include/bcm2711/irq.h</span></code>. I copied the |
| IRQ numbers from the datasheet and listed them all as macros with names. I also had to define the number of IRQS, which |
| was 216 in this case. The <code class="docutils literal notranslate"><span class="pre">MPID_TO_CORE(mpid)</span></code> macro was copied from another arm64 implementation.</p> |
| <div class="highlight-c notranslate"><div class="highlight"><pre><span></span><span class="cp">#define NR_IRQS 216</span> |
| <span class="cp">#define MPID_TO_CORE(mpid) (((mpid) >> MPIDR_AFF0_SHIFT) & MPIDR_AFFLVL_MASK)</span> |
| |
| <span class="cm">/* VideoCore interrupts */</span> |
| |
| <span class="cp">#define BCM_IRQ_VC_BASE 96</span> |
| <span class="cp">#define BCM_IRQ_VC(n) (BCM_IRQ_VC_BASE + n)</span> |
| |
| <span class="cp">#define BCM_IRQ_VC_TIMER0 BCM_IRQ_VC(0)</span> |
| <span class="cp">#define BCM_IRQ_VC_TIMER1 BCM_IRQ_VC(1)</span> |
| <span class="cm">/* More interrupts ... */</span> |
| </pre></div> |
| </div> |
| <p>Finally was to define the memory mapping within the <code class="docutils literal notranslate"><span class="pre">include/bcm2711/chip.h</span></code> file. I did so simply since I was only |
| testing on the 4GB version of the BCM2711. The RAM starts at address 0, and is roughly 4GB in size. 64 MB of that is |
| reserved for the memory-mapped I/O, so I had to be sure to remove that. I also defined the load address of the kernel in |
| memory for the chip.</p> |
| <div class="highlight-c notranslate"><div class="highlight"><pre><span></span><span class="cp">#define CONFIG_RAMBANK1_ADDR (0x000000000)</span> |
| |
| <span class="cm">/* Both the 4GB and 8GB ram variants use all the size in RAMBANK1 */</span> |
| |
| <span class="cp">#if defined(CONFIG_RPI4B_RAM_4GB) || defined(CONFIG_RPI4B_RAM_8GB)</span> |
| <span class="cp">#define CONFIG_RAMBANK1_SIZE GB(4) - MB(64)</span> |
| <span class="cp">#endif </span><span class="cm">/* defined(CONFIG_RPI4B_RAM_4GB) || defined(CONFIG_RPI4B_RAM_8GB) */</span> |
| |
| <span class="cm">/* Raspberry Pi 4B loads NuttX at this address */</span> |
| |
| <span class="cp">#define CONFIG_LOAD_BASE 0x480000</span> |
| </pre></div> |
| </div> |
| <p>The same load address had to be specified in the linker script for the Raspberry Pi 4B kernel. This scripts tells the |
| compiler how to lay out the kernel code in memory and what addresses to use. I was able to copy it from the PinePhone |
| and just change the load address to <code class="docutils literal notranslate"><span class="pre">0x480000</span></code>.</p> |
| </section> |
| <section id="figuring-out-the-boot"> |
| <h2>Figuring out the boot<a class="headerlink" href="#figuring-out-the-boot" title="Permalink to this heading"></a></h2> |
| <p>The first thing I wanted to do was determine how much work had already been done for aarch64 that would allow me to more |
| easily complete the port. In Lup’s blogs, he tested out support for his core type (ARM Cortex-A53 on the PinePhone) by |
| booting the aarch64 instance of QEMU with NuttX using that core. I decided to take the same approach, and was able to |
| successfully boot on ARM Cortex-A72 using QEMU following his blog. This was a nice confirmation that the hardware I was |
| using was already supported in NuttX for booting the OS and getting NSH working with a PL011 UART interface.</p> |
| <p>I cannot stress enough that the reason porting to this chip was made so much easier was because I am standing on the |
| shoulders of giants. NuttX contributors had already set up the boot scripts written in assembly, timer configuration, |
| interrupt handling and drivers for a lot of the standard features in aarch64 architectures. I did not have to deal with |
| any of this because of them, and it really cut down on the amount of assembly I had to read and understand. I also |
| barely had to write any assembly outside of debugging the boot process a little (we’ll get to that later). Not to |
| mention I had Lup’s well-written articles to guide me.</p> |
| <p>In order to compile and boot the board, I had to add a definition for <code class="docutils literal notranslate"><span class="pre">g_mmu_config</span></code>, which I was confused about and |
| left empty initially just to get past the compilation stage. I also defined the <code class="docutils literal notranslate"><span class="pre">GICR_OFFSET</span></code> and <code class="docutils literal notranslate"><span class="pre">GICR_BASE</span></code> macros |
| for the GICv2 interrupt controller by copying them from the Allwinner chip, which used the same controller. After |
| reading further in Lup’s blog, I learned that the boot script has a <code class="docutils literal notranslate"><span class="pre">PRINT</span></code> macro which is called early in the boot |
| process, and requires an implementation of <code class="docutils literal notranslate"><span class="pre">up_lowputc</span></code> to print to the console. This would be the first thing I need |
| to implement. This compiled, but when I booted the Pi, nothing happened.</p> |
| <p>After quite a while of trying different things and looking at other implementations, I noticed that many people were |
| using register manipulation directly in the early print functions. I decided I would do the same, but instead of |
| printing (a more complex operation), I would turn one of the GPIO pins high. I was able to measure this with my |
| multimeter and confirm that the GPIO did get set, so I knew that the <code class="docutils literal notranslate"><span class="pre">arm64_earlyprint_init</span></code> function was getting |
| called. Something was wrong with my UART configuration.</p> |
| <p>I then tried directly manipulating registers to put the text “hi” in the UART FIFO. When I booted again, this printed, |
| but then was followed by some garbled output. It appeared that the the <code class="docutils literal notranslate"><span class="pre">char</span> <span class="pre">*</span></code> pointer passed to the print function |
| was getting garbled. After troubleshooting by printing characters directly by calling my <code class="docutils literal notranslate"><span class="pre">arm64_lowputc</span></code> in the |
| assembly boot script, I discovered that I could print a string from the C definition if I declared the string as static. |
| I also investigated the elf generated by building and confirmed the string was located in <code class="docutils literal notranslate"><span class="pre">.rodata</span></code>. I was suspicious |
| that I was loading the kernel incorrectly into memory and some addresses were getting mixed up. Sure enough, I had |
| defined the load address in the linker script as <code class="docutils literal notranslate"><span class="pre">0x80000</span></code> instead of <code class="docutils literal notranslate"><span class="pre">0x480000</span></code>. Fixing this allowed me to see the |
| boot messages properly!</p> |
| <p>I received this message in the console:</p> |
| <div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">----gic_validate_dist_version: No GIC version detect</span> |
| <span class="go">arm64_gic_initialize: no distributor detected, giving up ret=-19</span> |
| <span class="go">_assert: Current Version: NuttX 12.6.0-RC0 6791d4a1c4-dirty Aug 4 2024 00:38:21 arm64</span> |
| <span class="go">_assert: Assertion failed panic: at file: common/arm64_fatal.c:375 task: Idle_Task process: Kernel 0x481418</span> |
| </pre></div> |
| </div> |
| <p>I had accidentally kept the GICv3 in my config files when copying things from other boards, and changed it to GICv2. |
| That resolved the issue and presented me with a new one:</p> |
| <div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">MESS:00:00:06.144520:0:----_assert: Current Version: NuttX 12.6.0-RC0 f81fb7a076-dirty Aug 4 2024 16:16:30 arm64</span> |
| <span class="go">_assert: Assertion failed panic: at file: common/arm64_fatal.c:375 task: Idle_Task process: Kernel 0x4811e4</span> |
| </pre></div> |
| </div> |
| <p>After enabling all of the debug output in the build options, this became:</p> |
| <div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">arm64_oneshot_initialize: cycle_per_tick 54000</span> |
| <span class="go">arm64_fatal_error: reason = 0</span> |
| <span class="go">arm64_fatal_error: CurrentEL: MODE_EL1</span> |
| <span class="go">arm64_fatal_error: ESR_ELn: 0xbf000002</span> |
| <span class="go">arm64_fatal_error: FAR_ELn: 0x0</span> |
| <span class="go">arm64_fatal_error: ELR_ELn: 0x48a458</span> |
| <span class="go">print_ec_cause: SError interrupt</span> |
| </pre></div> |
| </div> |
| <p>This looked like an unhandled interrupt, and after narrowing down which line was failing by adding log statements to the |
| kernel code, I discovered it was due to the spinlock code. An exception was being caused by the <code class="docutils literal notranslate"><span class="pre">ldaxr</span></code> instruction, |
| which the ARM documentation said could only be used once the MMU was enabled. I then enabled the MMU as well as its |
| debug information and was greeted with the lovely error:</p> |
| <div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">MESS:00:00:06.174977:0:----arm64_mmu_init: xlat tables:</span> |
| <span class="go">arm64_mmu_init: base table(L1): 0x4cb000, 64 entries</span> |
| <span class="go">arm64_mmu_init: 0: 0x4c4000</span> |
| <span class="go">arm64_mmu_init: 1: 0x4c5000</span> |
| <span class="go">arm64_mmu_init: 2: 0x4c6000</span> |
| <span class="go">arm64_mmu_init: 3: 0x4c7000</span> |
| <span class="go">arm64_mmu_init: 4: 0x4c8000</span> |
| <span class="go">arm64_mmu_init: 5: 0x4c9000</span> |
| <span class="go">arm64_mmu_init: 6: 0x4ca000</span> |
| <span class="go">init_xlat_tables: mmap: virt 4227858432x phys 4227858432x size 67108864x</span> |
| <span class="go">set_pte_table_desc:</span> |
| <span class="go">set_pte_table_desc: 0x4cb018: [Table] 0x4c4000</span> |
| <span class="go">init_xlat_tables: mmap: virt 0x phys 0x size 1006632960x</span> |
| <span class="go">set_pte_table_desc:</span> |
| <span class="go">set_pte_table_desc: 0x4cb000: [Table] 0x4c5000</span> |
| <span class="go">init_xlat_tables: mmap: virt 4718592x phys 4718592x size 192512x</span> |
| <span class="go">split_pte_block_desc: Splitting existing PTE 0x4c5010(L2)</span> |
| <span class="go">set_pte_table_desc:</span> |
| <span class="go">set_pte_table_desc: 0x4c5010: [Table] 0x4c6000</span> |
| <span class="go">init_xlat_tables: mmap: virt 4911104x phys 4911104x size 81920x</span> |
| <span class="go">init_xlat_tables: mmap: virt 4993024x phys 4993024x size 65536x</span> |
| <span class="go">enable_mmu_el1: MMU enabled with dcache</span> |
| <span class="go">nx_start: Entry</span> |
| <span class="go">up_allocate_heap: heap_start=0x0x4d3000, heap_size=0x47b2d000</span> |
| <span class="go">mm_initialize: Heap: name=Umem, start=0x4d3000 size=1202900992</span> |
| <span class="go">mm_addregion: [Umem] Region 1: base=0x4d32a8 size=1202900304</span> |
| <span class="go">arm64_fatal_error: reason = 0</span> |
| <span class="go">arm64_fatal_error: CurrentEL: MODE_EL1</span> |
| <span class="go">arm64_fatal_error: ESR_ELn: 0x96000045</span> |
| <span class="go">arm64_fatal_error: FAR_ELn: 0x47fffff8</span> |
| <span class="go">arm64_fatal_error: ELR_ELn: 0x489d28</span> |
| <span class="go">print_ec_cause: Data Abort taken without a change in Exception level</span> |
| <span class="go">_assert: Current Version: NuttX 12.6.0-RC0 96be557b64-dirty Aug 5 2024 14:56:42 arm64</span> |
| <span class="go">_assert: Assertion failed panic: at file: common/arm64_fatal.c:375 task: Idle_Task process: Kernel 0x481a34</span> |
| <span class="go">up_dump_register: stack = 0x4d2e10</span> |
| <span class="go">up_dump_register: x0: 0x13 x1: 0x4d32c0</span> |
| <span class="go">up_dump_register: x2: 0xfe215040 x3: 0xfe215040</span> |
| <span class="go">up_dump_register: x4: 0x0 x5: 0x0</span> |
| <span class="go">up_dump_register: x6: 0x1 x7: 0xdba53f65cc808a8</span> |
| <span class="go">up_dump_register: x8: 0xc4276feb17c016ba x9: 0xecbcfeb328124450</span> |
| <span class="go">up_dump_register: x10: 0xb7989dd7d34a1280 x11: 0x5ebf5f572386fdee</span> |
| <span class="go">up_dump_register: x12: 0x6f7c07d067f6e38 x13: 0x3f7b5adaf798b4d5</span> |
| <span class="go">up_dump_register: x14: 0xf3dffbe2e4cff736 x15: 0xd76b1c050c964ea0</span> |
| <span class="go">up_dump_register: x16: 0x6d6fa9cfeeb0eff8 x17: 0x1a051d808a830286</span> |
| <span class="go">up_dump_register: x18: 0x3f7b5adaf798b4bf x19: 0x4d3000</span> |
| <span class="go">up_dump_register: x20: 0x47fffff0 x21: 0x4d32d0</span> |
| <span class="go">up_dump_register: x22: 0x47b2cd30 x23: 0x4d32a8</span> |
| <span class="go">up_dump_register: x24: 0x4d32b0 x25: 0x4806f4</span> |
| <span class="go">up_dump_register: x26: 0x2f56f66b2df71556 x27: 0x74ee6bbfb5d438f4</span> |
| <span class="go">up_dump_register: x28: 0x7ef57ab47b85f74f x29: 0x9a7fa1cb06923003</span> |
| <span class="go">up_dump_register: x30: 0x489cf8</span> |
| <span class="go">up_dump_register:</span> |
| <span class="go">up_dump_register: STATUS Registers:</span> |
| <span class="go">up_dump_register: SPSR: 0x600002c5</span> |
| <span class="go">up_dump_register: ELR: 0x489d28</span> |
| <span class="go">up_dump_register: SP_EL0: 0x4d3000</span> |
| <span class="go">up_dump_register: SP_ELX: 0x4d2f40</span> |
| <span class="go">up_dump_register: TPIDR_EL0: 0x0</span> |
| <span class="go">up_dump_register: TPIDR_EL1: 0x0</span> |
| <span class="go">up_dump_register: EXE_DEPTH: 0x1</span> |
| </pre></div> |
| </div> |
| <p>Some more debugging allowed me to determine that the <code class="docutils literal notranslate"><span class="pre">CONFIG_RAM_START</span></code> and <code class="docutils literal notranslate"><span class="pre">CONFIG_RAM_SIZE</span></code> macros in the |
| defconfig for my nsh configuration were still set to the values from the PinePhone that I copied from. I set these to |
| the correct values for the Raspberry Pi 4B and got much further!</p> |
| <div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">MESS:00:00:06.211786:0:----irq_attach: In irq_attach</span> |
| <span class="go">irq_attach: before spin_lock_irqsave</span> |
| <span class="go">spin_lock_irqsave: me: 0</span> |
| <span class="go">spin_lock_irqsave: before spin_lock</span> |
| <span class="go">spin_lock: about to enter loop</span> |
| <span class="go">spin_lock: loop over</span> |
| <span class="go">spin_lock_irqsave: after spin_lock</span> |
| <span class="go">irq_attach: after spin_lock_irqsave</span> |
| <span class="go">irq_attach: before spin_unlock_irqrestore</span> |
| <span class="go">irq_attach: after spin_unlock_irqrestore</span> |
| <span class="go">arm64_serialinit: arm64_serialinit not implemented</span> |
| <span class="go">group_setupidlefiles: ERROR: Failed to open stdin: -38</span> |
| <span class="go">_assert: Current Version: NuttX 12.6.0-RC0 be262c7ad3-dirty Aug 5 2024 17:16:27 arm64</span> |
| <span class="go">_assert: Assertion failed : at file: init/nx_start.c:728 task: Idle_Task process: Kernel 0x48162c</span> |
| <span class="go">up_dump_register: stack = 0x4c0170</span> |
| <span class="go">up_dump_register: x0: 0x4c0170 x1: 0x0</span> |
| <span class="go">up_dump_register: x2: 0x0 x3: 0x0</span> |
| <span class="go">up_dump_register: x4: 0x0 x5: 0x0</span> |
| <span class="go">up_dump_register: x6: 0x3 x7: 0x0</span> |
| <span class="go">up_dump_register: x8: 0x4c7468 x9: 0x0</span> |
| <span class="go">up_dump_register: x10: 0x4c7000 x11: 0x4</span> |
| <span class="go">up_dump_register: x12: 0x4b8000 x13: 0x4b7000</span> |
| <span class="go">up_dump_register: x14: 0x1 x15: 0xfffffff7</span> |
| <span class="go">up_dump_register: x16: 0x48a654 x17: 0x0</span> |
| <span class="go">up_dump_register: x18: 0x1 x19: 0x0</span> |
| <span class="go">up_dump_register: x20: 0x4ac181 x21: 0x4bf430</span> |
| <span class="go">up_dump_register: x22: 0x0 x23: 0x4c0170</span> |
| <span class="go">up_dump_register: x24: 0x4c0170 x25: 0x2d8</span> |
| <span class="go">up_dump_register: x26: 0x240 x27: 0x4b7000</span> |
| <span class="go">up_dump_register: x28: 0xfdc3ed41d6862df6 x29: 0xbf8e8f7280a0100</span> |
| <span class="go">up_dump_register: x30: 0x481bf8</span> |
| <span class="go">up_dump_register:</span> |
| <span class="go">up_dump_register: STATUS Registers:</span> |
| <span class="go">up_dump_register: SPSR: 0x20000245</span> |
| <span class="go">up_dump_register: ELR: 0x480230</span> |
| <span class="go">up_dump_register: SP_EL0: 0x4c7000</span> |
| <span class="go">up_dump_register: SP_ELX: 0x4c6e90</span> |
| <span class="go">up_dump_register: TPIDR_EL0: 0x4bf430</span> |
| <span class="go">up_dump_register: TPIDR_EL1: 0x4bf430</span> |
| <span class="go">up_dump_register: EXE_DEPTH: 0x0</span> |
| <span class="go">dump_tasks: PID GROUP PRI POLICY TYPE NPX STATE EVENT SIGMASK STACKBASE STACKSIZE USED FILLED COMMAND</span> |
| <span class="go">dump_tasks: ---- --- --- -------- ------- --- ------- ---------- ---------------- 0x4c4000 4096 144 3.5% irq</span> |
| <span class="go">dump_task: 0 0 0 FIFO Kthread - Running 0000000000000000 0x4c5010 8176 1200 14.6% Idle_Task</span> |
| |
| <span class="go">CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.9 | VT102 | Offline | ttyUSB0</span> |
| </pre></div> |
| </div> |
| <p>We actually got into tasks now! It appears stdin failed to open because in my Mini-UART driver implementation I had the |
| <code class="docutils literal notranslate"><span class="pre">attach</span></code> and <code class="docutils literal notranslate"><span class="pre">ioctl</span></code> functions return <code class="docutils literal notranslate"><span class="pre">-ENOSYS</span></code>. Just changing this to 0 for success in the interim allowed us to |
| get even further, and I could see the beginnings of NSH spawning.</p> |
| <div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">mm_initialize: Heap: name=Umem, start=0x4cc000 size=4222828544</span> |
| <span class="go">mm_addregion: [Umem] Region 1: base=0x4cc2a8 size=4222827856</span> |
| <span class="go">mm_malloc: Allocated 0x4cc2d0, size 144</span> |
| <span class="go">mm_malloc: Allocated 0x4cc360, size 80</span> |
| <span class="go">gic_validate_dist_version: GICv2 detected</span> |
| <span class="go">up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 54.0MHz</span> |
| <span class="go">arm64_oneshot_initialize: oneshot_initialize</span> |
| <span class="go">mm_malloc: Allocated 0x4cc3b0, size 48</span> |
| <span class="go">arm64_oneshot_initialize: cycle_per_tick 54000</span> |
| <span class="go">uart_register: Registering /dev/console</span> |
| <span class="go">mm_malloc: Allocated 0x4cc3e0, size 80</span> |
| <span class="go">mm_malloc: Allocated 0x4cc430, size 80</span> |
| <span class="go">uart_register: Registering /dev/ttys0</span> |
| <span class="go">mm_malloc: Allocated 0x4cc480, size 80</span> |
| <span class="go">mm_malloc: Allocated 0x4cc4d0, size 80</span> |
| <span class="go">mm_malloc: Allocated 0x4cc520, size 80</span> |
| <span class="go">mm_malloc: Allocated 0x4cc570, size 32</span> |
| <span class="go">mm_malloc: Allocated 0x4cc590, size 64</span> |
| <span class="go">work_start_highpri: Starting high-priority kernel worker thread(s)</span> |
| <span class="go">mm_malloc: Allocated 0x4cc5d0, size 336</span> |
| <span class="go">mm_malloc: Allocated 0x4cc720, size 8208</span> |
| <span class="go">nxtask_activate: hpwork pid=1,TCB=0x4cc5d0</span> |
| <span class="go">nx_start_application: Starting init thread</span> |
| <span class="go">task_spawn: name=nsh_main entry=0x48b24c file_actions=0 attr=0x4cbfa0 argv=0x4cbf98</span> |
| <span class="go">mm_malloc: Allocated 0x4ce730, size 1536</span> |
| <span class="go">mm_malloc: Allocated 0x4ced30, size 64</span> |
| <span class="go">mm_malloc: Allocated 0x4ced70, size 32</span> |
| <span class="go">mm_malloc: Allocated 0x4ced90, size 8208</span> |
| <span class="go">nxtask_activate: nsh_main pid=2,TCB=0x4ce730</span> |
| <span class="go">lib_cxx_initialize: _sinit: 0x4ad000 _einit: 0x4ad000</span> |
| <span class="go">mm_malloc: Allocated 0x4d0da0, size 848</span> |
| <span class="go">mm_free: Freeing 0x4d0da0</span> |
| <span class="go">mm_free: Freeing 0x4ced70</span> |
| <span class="go">mm_free: Freeing 0x4ced30</span> |
| <span class="go">nxtask_exit: nsh_main pid=2,TCB=0x4ce730</span> |
| <span class="go">mm_free: Freeing 0x4ced90</span> |
| <span class="go">mm_free: Freeing 0x4ce730</span> |
| <span class="go">nx_start: CPU0: Beginning Idle Loop</span> |
| </pre></div> |
| </div> |
| <p>It seemed like we were waiting on an interrupt which never occurred. This was weird, because my Mini-UART driver had an |
| interrupt implementation and appeared to be written just fine. This took hours of debugging, logging from interrupt |
| handlers and dumping register values, but eventually I determined that the BCM2711 datasheet actually had an error where |
| the TX and RX interrupt fields were swapped in the datasheet. A blog post online had mentioned this for the BCM2835, but |
| it appeared to be an issue on this chip as well. Now we were booting into NSH!</p> |
| <p>It was at this point that the port is considered a success, since I was able to boot into NSH and successfully run the |
| <code class="docutils literal notranslate"><span class="pre">ostest</span></code> benchmark. I went on to write the start of a few more drivers, like the GPIO driver, but this completed the |
| requirements for an initial port and is most of what ended up being submitted in the initial pull request.</p> |
| </section> |
| </section> |
| |
| |
| </div> |
| </div> |
| <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer"> |
| <a href="../port_relatedkernelconfigrations.html" class="btn btn-neutral float-left" title="The list of related kernel configurations" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a> |
| <a href="port_arm_cm4.html" class="btn btn-neutral float-right" title="The case of ARM CM4 & cxd32xx @NuttX12.4.0" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a> |
| </div> |
| |
| <hr/> |
| |
| <div role="contentinfo"> |
| <p>© Copyright 2023, The Apache Software Foundation.</p> |
| </div> |
| |
| |
| |
| </footer> |
| </div> |
| </div> |
| </section> |
| </div> |
| <script> |
| jQuery(function () { |
| SphinxRtdTheme.Navigation.enable(true); |
| }); |
| </script> |
| |
| </body> |
| </html> |
