Daffodil-VSCode Developer's Guide

Build Status

CI nightly tests

Table of Contents

Welcome

This guide contains developer-oriented instructions on how to setup your local development environment for contributing to this project.

If you would like the latest stable release of the extension, please reference the README file.

The project is currently growing, please refer to the wiki for an overview of what the extension includes. There is also user documentation on the right pane with release specific use case guides.

Prerequisites

Install Git

If your system doesn’t have Git installed, install Git. Select the appropriate operating system and follow the listed instructions. To verify git is installed on your system, enter git -v into your system’s terminal and it should display the version of git if properly installed.

Install Visual Studio Code (VSCode)

Install VSCode per official VSCode documentation:

Installing Build Requirements

This section will provide describe what requirements are needed to build with a step-by-step guide to assist developers of varying skill levels.

Summary of Build Requirements

  • Java Development Kit (JDK) between 8 and 17
  • SBT 0.13.8 or higher
  • Node 20 or higher
  • Yarn Classic

Step-by-step Guide for Installing Build Requirements

Installing Node

Navigate to the Node Download page. Select the instructions for installing the latest LTS version of Node.js for your operating system using fnm with npm.

Picture1

Installing Node on Windows Note

Make sure you have winget installed. If it’s not installed, you can install it by typing in Install-Module -Name Microsoft.WinGet.Client into PowerShell. After following all the steps outlined on Node’s page, you want to create a PowerShell profile which is a PowerShell script that gets executed every time a new instance of PowerShell is opened. Then append fnm env --use-on-cd --shell powershell | Out-String | Invoke-Expression per fnm's guidance.

Installing Java (JDK 17)

You can install Oracle's JDK 17 or alternatively install the OpenJDK equivalent.

Build Issues with Higher JDK Versions

Note that higher versions of the JDK may cause extension building issues and is not advised.

Switching Java Versions on Linux

You might have Java already installed. You can change the default Java provider and version by running sudo update-alternatives --config java.

Installing SBT

Install SBT for your appropriate operating system:

Enabling Yarn from Within Node

Do Not Use the Latest Version of Yarn

Do not upgrade versions of Yarn for repository. Do not follow the instructions on Yarn’s official website. This will break the extension packaging process.

How to Enable

Type into the console corepack enable yarn.

Contributing and Setup

Forking the Project

This is an Apache Project. If you would like to contribute, you will need to fork the daffodil-vscode main branch to your own repo of choice and create pull requests to the main branch with your code changes. image

Cloning the Project to Local Environment

Once forked, you can clone that forked repository to your own local environment. This can be done by using git clone, followed by copy and pasting the https or SSH urls shown under the “Code” dropdown. We recommend using SSH. Instructions for setting up SSH can be found in Setting up SSH Keys.

image

image

Once cloned, you can now create branches, commits, and push changes back to your remote fork. You may make changes with any IDE, but we highly encourage using VSCode for testing for maximum compatability.

Setting up SSH Keys

Enter ssh-keygen into your terminal to generate an RSA key. Follow the prompts for naming the public and private key files (optional) and giving the keys an optional passphrase.

The public and private keys are stored in a folder called .ssh in your users folder. The public key file is denoted with a .pub whereas the private key doesn’t have a file extension.

Navigate to SSH and GPG keys GitHub settings. Log into your GitHub account if needed. Click on New SSH Key.

ssh_keys

Give your new SSH Key a name and paste the contents of the .pub file into the key textbox

add_new_ssh_key

Note that the above image has parts of the key blurred out for confidentiality.

Click on Add SSH key and follow GitHub prompts for verification.

GitHub has their own guide on setting up SSH keys which can be alternatively followed.

General Workflow

Ensure that you keep your fork synced with the daffodil-vscode main by using the sync fork button, this ensures that you are developing with up to date code, so that you can be sure your changes work with present code.

image

If any changes were made while you were working on yours, you will need to pull these changes down to your local environment and merge them with your changes before pushing back to remote.

Once changes are pushed, you can make pull requests with completed changes back to the main daffodil-vscode branch. You can use the contribute drop down to create a pull request back to the main branch.

image

Pull Request Approval and Merge Process

Once you‘ve created a pull request to the main daffodil-vscode branch, it must go through the Apache Daffodil project’s approval process before it can be merged.

Approval Requirements

All pull requests require at least two +1 votes (green approvals) from PMC members or committers before they can be merged. This ensures that changes are properly reviewed and meet the project's quality standards. The approvals must come from committers with write access to the repository.

Review Cycle

The review process typically follows this workflow:

  1. Submit Pull Request: Create your PR with your proposed changes to the main branch
  2. Reviewer Feedback: PMC members, committers, and other contributors will review your code and provide feedback
  3. Address Feedback: Add new commits to address review comments
    • Important: Do not squash previously reviewed commits during the review cycle
    • Each review iteration should involve a separate, distinct commit to preserve the review commentary
    • The pull request will automatically update with new commits pushed to your branch
  4. Approval: Once reviewers are satisfied, they will provide +1 votes (approvals)
  5. Merge: After receiving at least two +1 votes from PMC members or committers, your PR is ready to be merged. Squash all of your commits on your branch into one commit before the final merge, as our PR pipeline has a check that only allows for PRs with a single commit

Merging Process

Only committers with merged GitHub and ASF (Apache Software Foundation) accounts have the authority to merge pull requests.

Merge Method: When merging, committers must:

  • Click the “Merge pull request” dropdown
  • Select “Rebase and merge”
  • Do not use “Squash and merge” or “Create a merge commit”

This maintains a clean, linear commit history while preserving individual commits from the pull request.

Post-Merge Actions

After a pull request is merged, the committer who performed the merge should mark the associated GitHub issue as “Closed” and/or move it to the appropriate project board status.

Before Starting Work

To signal to the community that you are actively working on an issue and avoid duplicate efforts:

  • Assign the GitHub issue to yourself before beginning development
  • Add a comment on the issue to let others know you're working on it
  • If the issue is part of a milestone project board, move it to “In Progress”

Additional Resources

The Apache Daffodil Code Contributor Workflow documentation provides additional context about the broader Apache Daffodil contribution process. While the daffodil-vscode workflow is similar in many respects, there are some differences specific to this project, such as the merge method and commit requirements described above.

Opening the Repository in VSCode

There are multiple ways of opening the repository that you have cloned in VSCode:

  1. In a terminal, type in code <PATH TO THE CLONED REPOSITORY>
  2. Clicking on open folder at the VSCode homescreen

open_folder_vscode

  1. At the top options bar, File -> Open Folder

file-open-folder

Recommended VSCode Extensions

Upon opening, VSCode may prompt you to install recommended extensions. Below are the following reasons why you may need to install the recommended extensions.

Verifying Setup Can Build

Navigate inside your cloned folder in a terminal. Enter yarn to install required packages. If it prompts you to install yarn 1.22.XX, type y.

Then type in yarn package. If there’s new .vsix file in the folder, then you have successfully set up your development environment correctly. For more information about yarn package, read Yarn Package.

vsix

Lastly, run yarn test. All tests should pass without any errors. More information can be found under Automated Testing Suite (Yarn Test)

yarn_test_succ_output

Alternatively, you can run all of the commands in a single line by running yarn && yarn package && yarn test && echo "All good!".

Yarn Package

If you would like to confirm that your changes compile, you can run the extension through the VSCode debugger as shown below. Under the run and debugger, you should see a launch.json already loaded for the extension, just hit the green play button. This will open a new debug window of vscode that will have the extension built and running. You can then test any changes made and ensure it is operating as intended. For more information on the debugging in VSCode, please reference Debugging the Extension section.

image

You can also do this manually on the CLI with the following command: yarn package

This command will perform the following tasks:

  • Create sbt zip package and unzip it
  • Install dependencies
  • Compiles Extension
  • Packages Extension
  • Creates File
    • apache-daffodil-vscode-*.vsix

You can then take this .vsix file and install the extension into your vscode instance. Be sure you don’t already have the extension installed from the marketplace or the versioning could cause issues with seeing changes from your build.

image

Automated Testing Suite

The Apache Daffodil VS Code Extension comes with an automated test suite. Run it as follows:

yarn test
Testing Against a Specific Version of VS Code

By default, the test suite will use the earliest supported release of VS Code. To test against any specific version of VS Code (in this example, VS Code version 1.74.3), execute the test suite as follows, setting DAFFODIL_TEST_VSCODE_VERSION to the desired version:

DAFFODIL_TEST_VSCODE_VERSION=1.74.3 yarn test

Set DAFFODIL_TEST_VSCODE_VERSION to stable to use the latest stable release, or to insiders to use the latest (nightly) insiders build.

Debugging the Extension

Create a sampleWorkspace folder in the folder one level higher than where daffodil-vscode currently resides. For example, if you have your daffodil-vscode folder stored in a folder called repos, then make a folder in repos called sampleWorkspace.

sampleworkspace-loc

It’s advised to copy sample data files and sample DFDL schemas (.dfdl.xsd) in here. You can find DFDL schemas at DFDL Schemas for Commercial and Scientific Data Formats.

Next start debugging the extension. Make sure the “Extension” debug configuration is currently selected. You can press the green run button or alternatively press F5.

debug_loc

Starting the debugging session creates a terminal inside of VSCode running yarn watch and builds the extension. yarn watch automatically recompiles changes to the code in real-time. It also displays problems that occur in the code in real-time. For the changes to take effect however, you will have to stop and restart debugging.

You may see this window pop up

webpack_not_fully_loaded

Click on debug anyway once yarn watch says vite is fully compiled.

webpack-fully-loaded

A new VSCode window should’ve popped up with the sampleWorkspace opened.

debug-window

Open the command palette with Ctrl + Shift + P and type in Daffodil Debug. Select the Configure launch.json option.

config-launc-josn-command-pal

Your window should look like this.

daf-debug-settings

Scroll down and check these following options

open_3_checkbox

Click save

bottom-daf-debug

Click on the Run and Debug Icon. Wizard Config should show at the top.

wiz-config

Then select a schema file and a data file. (Note, if you want to hard code where the files are, you’re able to change the DFDL and data file path in the Daffodil Configure launch.json window)

select-dfdl-schema

select-test-image

Your window should look like the following. Note that you may have to move some tabs around.

ide-debug

Here’s an example view of the data editor (OmegaEdit) with the schema on the left.

ide-debug2

Here’s the infoset diff view. The infoset is the resulting output XML/JSON file.

ide-debug4

Changing the sampleWorkspace folder name and location

If you would like to specify a different name and/or location for the sampleWorkspace, modify the following line in .vscode/launch.json:

change-sampleworkspace-name

Test a Local Version of Apache Daffodil Debugger

The local Apache Daffodil™ Extension for Visual Studio Code downloads and caches the Apache Daffodil™ Debugger corresponding to the latest extension release. If you want to test a local version of the Apache Daffodil Debugger, you need to:

  • add "useExistingServer": true to the configuration in your launch.json in the sample workspace;

  • launch the backend debugger locally, using a launch configuration like below:

    {
  "type": "scala",
  "name": "DAPodil",
  "request": "launch",
  "mainClass": "org.apache.daffodil.debugger.dap.DAPodil",
  "args": []
}

    This will start the debug adapter and await a connection from the Apache Daffodil VS Code Extension (usually on TCP port 4711); and

  • debug your schema file, as long as your launch.json has the useExistingServer setting above.

Debugging UI Source

Inspecting the HTML

To inspect the HTML, open up the UI window (e.g., Configure launch.json, Data Editor) that you wish to inspect and open the Webview Developer Tools with Ctrl + Shift + P and typing Developer: Open Webview Developer Tools.

NOTE: This only debugs the UI from the DOM. It does not add any debugging functionality within the extension's JS/TS source files.

ui-debug1

From the Elements tab, you can inspect the HTML by expanding and hovering elements to highlight them in the UI.

ui-debug2

Likewise, you can use the Inspect tool to click on items in the UI to see the corresponding HTML element.

ui-debug3

ui-debug4

Debugging the JavaScript

To debug the JavaScript, add a debugger statement to the source code where you want to debug. The debugger statement acts like a breakpoint and pauses execution when reached.

ui-debug5

Start the Extension debug window and open the UI window (e.g., Configure launch.json, Data Editor) that you are debugging, and open the Webview Developer Tools with Ctrl + Shift + P and typing Developer: Open Webview Developer Tools. Then, complete the action necessary to reach the debugger statement if need be. After the debugger statement is reached, the execution will pause, and the source will be shown in the Webview Developer Tools window.

ui-debug6

You can then add breakpoints by clicking the line number.

ui-debug7

You can also use the bar on the right side to continue or step through the execution.

ui-debug8

To examine the value of a variable while stepping through the execution, you can add a watch expression in the right pane.

ui-debug9

ui-debug10

Troubleshooting

You may run into issues with building, running tests, or debugging the extension. The follow sections will describe some issues you may encounter and discuss remedies.

Yarn Package Issues

If Yarn Keeps Updating to The Latest Version

As of typing this document (Feb 2025), the latest version of yarn is 4.6.0. If you type yarn and your console outputs the following or anything similar:

➤ YN0087: Migrated your project to the latest Yarn version ��
➤ YN0000: · Yarn 4.6.0
➤ YN0000: ┌ Resolution step
➤ YN0085: │ + @omega-edit/client@npm:0.9.83, @tsconfig/svelte@npm:5.0.2, @types/glob@npm:8.1.0, @types/mocha@npm:10.0.6, @types/node@npm:20.11.30, @types/vscode-webview@npm:1.57.4, @types/vscode@npm:1.95.0, @viperproject/locate-java-home@npm:1.1.15, @vscode/debugadapter-testsupport@npm:1.65.0, @vscode/debugadapter@npm:1.67.0, @vscode/test-electron@npm:2.3.8, @vscode/vsce@npm:2.22.0, @vscode/webview-ui-toolkit@npm:1.4.0, await-notify@npm:1.0.1, chai@npm:4.4.1, and 703 more.
➤ YN0000: └ Completed in 9s 958ms
➤ YN0000: ┌ Post-resolution validation
➤ YN0002: │ apache-daffodil-vscode@workspace:. doesn't provide react (pa7c88), requested by @vscode/webview-ui-toolkit.
➤ YN0086: │ Some peer dependencies are incorrectly met by your project; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code.
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0013: │ 696 packages were added to the project (+ 397.25 MiB).
➤ YN0000: └ Completed in 4m 47s
➤ YN0000: ┌ Link step
➤ YN0007: │ esbuild@npm:0.19.9 must be built because it never has been before or the last one failed
➤ YN0007: │ svelte-preprocess@npm:5.1.1 [5fe27] must be built because it never has been before or the last one failed
➤ YN0007: │ svelte-preprocess@npm:5.1.1 [ab741] must be built because it never has been before or the last one failed
➤ YN0007: │ keytar@npm:7.9.0 must be built because it never has been before or the last one failed
➤ YN0007: │ protobufjs@npm:7.4.0 must be built because it never has been before or the last one failed
➤ YN0000: └ Completed in 2m 40s
➤ YN0000: · Done with warnings in 7m 37s

this means that there is a yarn project that is initialized in a folder that’s higher up from the cloned repository folder. This will negatively affect the extension packaging process. A remedy for the issue is to remove all yarn files and the package.json file in the higher folder that the yarn project is initialized in.

To remedy this, you need to change versions of Yarn. Use yarn set version 1.22.22 to change versions of Yarn. Documentation for set-version.

Yarn Test Issues

Data Editor Opens Test Case Failing

This means port 9000 is Occupied. See the current workarounds section in “data editor opens” test fails if Port 9000 is Occupied · Issue #1175 · apache/daffodil-vscode.

TLS Certificate Issues

HTTPS TLS certificates are verified by default. When running the test suite in certain environments (e.g., company VPN that uses endpoint protection), TLS certificate verifications may fail with a self-signed certificate error. If this is the case, either have node trust the endpoint protection certificate, or use one of these workarounds to disable the certificate verification:

NODE_TLS_REJECT_UNAUTHORIZED=0 yarn test

or

node ./out/tests/runTest.js --disable_cert_verification

WARNING: Do not export NODE_TLS_REJECT_UNAUTHORIZED=0 into your environment as it will disable TLS certificate verification on all node HTTPS connections done in that shell session.

Tests Failing When Connected Through SSH

Unit tests may not work when connected through SSH.

Debugging Issues

SELinux Port Functionality

If you’re running into frequent issues with connectivity or VSCode freezing, it may be worth it to disable SELInux enforcing mode. To check to see if SELinux OS is in enforcing mode, you can type getenforce in a console. If it outputs Enforcing, you’ll want to set it to Permissive by using sudo setenforce Permissive.

Windows Yarn Test Window Not Loading Extensions

If you run yarn test, and are running into an issue where the test window is saying there’s a missing dependent extension

missing-ext

and yarn tests under the getCommands section are failing,

getCommands-test-fail

Inside of src\tests\runTest.ts, replace

      {
        encoding: 'utf-8',
        stdio: 'inherit',
      }

with

      {
        encoding: 'utf-8',
        stdio: 'inherit',
        shell: os.platform().toLowerCase().startsWith('win'),
      }

and add import os from 'os' near the top of the file along with the other import statements.

Variable Values Not Displaying in VSCode Debugger

When debugging in VSCode, if you‘re noticing that it’s not possible to view variables' values or they‘re appearing as uncaught references when they aren’t supposed to, as shown below,

image

the yarn script "vite:dev", under "scripts" in package.json, should be utilized for building the extension for debugging capabilities.

Using Symlink Paths

Another important issue to note when choosing working directories is that VSCode, like many IDEs, seems to not like projects that are on paths reachable via symlinks. Using symlinks in paths is highly likely to cause problems and thus should be avoided.

Re-trying Verification After Errors

Type in git clean -fdx. Then run yarn && yarn package && yarn test && echo "All good!". If issues persist, you may want to uninstall Node and reinstall it. If that doesn’t remedy the issue, you may have to create a fresh VM.

Inspecting the Log Files

For issues not mentioned in this section, the log files for the debugger are located in your OS temp directory (e.g. /tmp for Ubuntu 24.04.2). The log file for the debugger should be named daffodil-debugger-<port>.log when debugging the extension or yarn-test-daffodil-debugger-<port>.log when running yarn test.

The log files for the data editor are located in your OS app data path (e.g. /home/<user>/.local/share/omega_edit). The log files for the data editor should be named serv-<port>.log for the server logs and dataEditor-<port>.log for the data editor logs when using the data editor or test-serv-<port>.log and test-dataEditor-<port>.log when running yarn test.

Development Specific

Dependency Licensing Information

The extension has a pipeline for ensuring that yarn packages and other relevant dependencies are in compliance with accepted licenses. Licenses that are not compliant or don't have their licenses noted in the repository will be flagged in CI.

Dependencies' licensing information is compiled under ./build/package. Contents of the dependencies' LICENSE, NONOTICE, NOTICE files are to be appended to the corresponding file in following the existing licensing file's format. Refer to the prior appended entries as a guideline for the formatting.

Developers may have to modify contents of the files under ./build/package in events where dependencies may be affected, such as package updates, additions, etc.

Building the Documentation

Install Pandoc

Before running commands to build the documentation, be sure to install Pandoc.

Command to build the Documentation

To build docx (Word formatted) documentation, from the top of the cloned repository, run:

cd docs && make all

Adding Release Candidate Information to Apache Daffodil Site

On every release candidate the following process needs to be done on the apache/daffodil-site:

  • Create a new file under site/_vscode with the release version as the name, e.g. 1.4.1.md using the following example. Look at the closed tickets for the milestone version to add bullets to the New Features & Fixes section.
---

title: <version>
release: rc1 (replace with current rc version or "final" if rc moved to final release)
apache: true
date: <date-rc-was-created-and-put-out-for-vote>
summary: >
    <summary of release candidate>

source-dist:
    - "apache-daffodil-vscode-<version>-src.zip"

binary-dist:
    - "apache-daffodil-vscode-<version>.vsix"
---

This release contains new features and fixes listed below.

# New Features & Fixes

* <category>: (Debugger, Data Editor, etc.)
    * <bulleted list of features/fixes>
* <category>: (Debugger, Data Editor, etc.)
    * <bulleted list of features/fixes>
* Known Issues:
    * <bulleted list of known issues>

# Closed Issues

[GitHub v<version> Closed Issues](<link to milestone's closed issues>)
  • Create a PR to the daffodil-site repository
    • After each RC and and when an RC is promoted to final release, the file needs to be updated to keep track of where the release process is.
    • Once that initial PR is merged that release page will not be shown unless the value released: is set to final which is done after promoting a RC to final release.

Reviewing and Verifying Dependency Bot Updates

For GitHub CI action updates (pull requests that start with Bump actions/...), make sure the affected workflows still operate as expected (they are automatically CI tested). GitHub CI actions update workflow YAML files, and are part of the CI infrastructure and not a code dependency. These should be relatively quick and easy to assess compared to code dependencies.

If the updates are not GitHub CI action updates, then additional scrutiny is required. When reviewing and verifying dependency bot updates that are part the software supply chain being distributed, please use the following checklist:

  • [ ] Do all automated continuous integration checks pass?
  • [ ] Is the update a patch, minor, or major update?
  • [ ] Is the license still compatible with ASF License Policy?
  • [ ] Have any changes been made to LICENSE/NOTICE files that need to be incorporated?
  • [ ] Have any transitive dependencies been added or changed?

Testing Information

Testing for this extension comprises of unit testing, CI/CD tests for pull requests (PRs), and manual tests.

List of manual tests can be found in the testing README.

Monitoring Project Status

Milestone-level project status can be monitored using the Projects tab in the Project's GitHub repository.

DFDL Intellisense Suggestion Cross Reference

Documents exist that attempt to map the DFDL schema elements and attributes in the Apache DFDL Specification that can be used as a guide to determine the intellisense suggestions that the extension should make based on the element or attribute marked by the cursor when intellisense is triggered. The documents include an excel spreadsheet, markdown file exports of the spreadsheets, and html exports of the spreadsheets. These files can be found in the source code at src/language/dfdl-xref and src/language/dfdl-xref/html.

DFDL Schema Definition Error Listing

A document exists that shows possible schema definition errors that can occur when developing DFDL schemas and the conditions that create those errors. This file can be found in the source code at src/language/dfdl-xref/DFDL_Schema_Defintion_Errors.md.

Thank you for your interest in contributing to this project

You can ask questions on the dev@daffodil.apache.org or users@daffodil.apache.org mailing lists. You can report bugs via GitHub Issues.