Google Git
Sign in
apache / geode / 495f3b0cffe0d0200521c266ea26c9e53cf6d629
commit495f3b0cffe0d0200521c266ea26c9e53cf6d629[log] [tgz]
authorKirk Lund <klund@apache.org>Thu Jun 02 16:36:05 2022 -0700
committerGitHub <noreply@github.com>Thu Jun 02 16:36:05 2022 -0700
tree3bfbb9acf88133597432d5fa5d6236cc4727ee31
parente0aac2090155dd03bccbe0195997801f8db80b1f [diff]
GEODE-10327: Overhaul GfshRule to kill processes and save artifacts (#7758)

PROBLEM

Tests that use GfshRule leave behind orphaned processes and do not save
artifacts for debugging failures.

SOLUTION

GfshRule needs to cleanup all processes it forks. It also needs to save
off all runtime artifacts such as logging, stats, pid files, diskstores
to enable debugging of test failures.

DETAILS

Enhance GfshRule and modify all tests using it for proper debugging and
to prevent test pollution.

Overhaul of GfshRule:

* kill ALL geode processes during cleanup
* use FolderRule to ensure all logs and files are properly saved off
when a test fails
* extract GfshExecutor from JUnit rule code
* GfshExecutor allows a test to use any number of Geode versions with
just one GfshRule
* add Gfsh log level support for easier debugging
* add support for new VmConfiguration to allow control over Geode and
Java versions
* overhaul API of GfshRule and companion classes for better consistency
and design

New FolderRule:

* replaces TemporaryFolder and saves off all content when a test fails
* creates root directory under the gradle worker instead of under temp

Update HTTP session caching module tests:

* use new FolderRule to save all artifacts when a test fails
* use nio Paths for filesystem variables

Update acceptance and upgrade tests that use GfshRule:

* use new improved GfshRule and GfshExecutor
* use new FolderRule instead of TemporaryFolder to save all artifacts
when a test fails
* use --disable-default-server in tests with no clients
* fix flakiness of many tests by using random ports instead of default
or hardcoded port values
* reformat GfshRule API usage in tests to improve readability and
consistency
* add GfshStopper to provide common place to await process stop (stop
locator/server is async so restarting with same ports is very prone
to hitting BindExceptions)

Update ProcessUtils:

* extract NativeProcessUtils and make it public for direct use
* rename InternalProcessUtils as ProcessUtilsProvider and move to its
own class
* rethrow IOExceptions as UncheckedIOExceptions
* fix flakiness in NativeProcessUtilsTest by moving findAvailablePid
into test method

Minor changes:

* improve code formatting and readability
* convert from old io File to nio Path APIs as much as possible
* close output streams to fix filesystem issues on Windows

Fixes flaky test tickets:

* DeployJarAcceptanceTest GEODE-9615
* possibly other tests that uses GfshRule

Changes for resubmit:

* log error message if unable to delete folder
* keep default constructor on GfshRule
* ensure IO streams have proper error handling and don't cause failures
on windows

Changes to build pipelines:

* make jdk17 tests gating

NOTES

The labels jdk8, jdk17, windows, windows-jdk8 and windows-jdk17 were 
used to run tests on more environments.

This PR contains mostly test and framework changes. The only product
code altered is ServerLauncher and several classes in
org.apache.geode.internal.process, all of which is in geode-core.
97 files changed
tree: 3bfbb9acf88133597432d5fa5d6236cc4727ee31
  1. .github/
  2. boms/
  3. build-tools/
  4. ci/
  5. dev-tools/
  6. docker/
  7. etc/
  8. extensions/
  9. geode-assembly/
  10. geode-book/
  11. geode-common/
  12. geode-concurrency-test/
  13. geode-connectors/
  14. geode-core/
  15. geode-cq/
  16. geode-deployment/
  17. geode-docs/
  18. geode-dunit/
  19. geode-gfsh/
  20. geode-http-service/
  21. geode-jmh/
  22. geode-junit/
  23. geode-log4j/
  24. geode-logging/
  25. geode-lucene/
  26. geode-management/
  27. geode-membership/
  28. geode-memcached/
  29. geode-old-client-support/
  30. geode-old-versions/
  31. geode-pulse/
  32. geode-rebalancer/
  33. geode-serialization/
  34. geode-server-all/
  35. geode-tcp-server/
  36. geode-unsafe/
  37. geode-wan/
  38. geode-web/
  39. geode-web-api/
  40. geode-web-management/
  41. gradle/
  42. static-analysis/
  43. .asf.yaml
  44. .gitattributes
  45. .gitignore
  46. build.gradle
  47. BUILDING.md
  48. CODE_OF_CONDUCT.md
  49. CODEOWNERS
  50. CODEWATCHERS
  51. COMMITWATCHERS
  52. gradle.properties
  53. gradlew
  54. gradlew.bat
  55. KEYS
  56. LICENSE
  57. NOTICE
  58. README.md
  59. settings.gradle
  60. TESTING.md
README.md

Apache Geode logo

Build Status License Maven Central homebrew Docker Pulls Total alerts Language grade: Java Language grade: JavaScript Language grade: Python

Contents

  1. Overview
  2. How to Get Apache Geode
  3. Main Concepts and Components
  4. Location of Directions for Building from Source
  5. Geode in 5 minutes
  6. Application Development
  7. Documentation
  8. Wiki
  9. How to Contribute
  10. Export Control

Overview

Apache Geode is a data management platform that provides real-time, consistent access to data-intensive applications throughout widely distributed cloud architectures.

Apache Geode pools memory, CPU, network resources, and optionally local disk across multiple processes to manage application objects and behavior. It uses dynamic replication and data partitioning techniques to implement high availability, improved performance, scalability, and fault tolerance. In addition to being a distributed data container, Apache Geode is an in-memory data management system that provides reliable asynchronous event notifications and guaranteed message delivery.

Apache Geode is a mature, robust technology originally developed by GemStone Systems. Commercially available as GemFire™, it was first deployed in the financial sector as the transactional, low-latency data engine used in Wall Street trading platforms. Today Apache Geode technology is used by hundreds of enterprise customers for high-scale business applications that must meet low latency and 24x7 availability requirements.

How to Get Apache Geode

You can download Apache Geode from the website, run a Docker image, or install with Homebrew on OSX. Application developers can load dependencies from Maven Central.

Maven

<dependencies>
    <dependency>
        <groupId>org.apache.geode</groupId>
        <artifactId>geode-core</artifactId>
        <version>$VERSION</version>
    </dependency>
</dependencies>

Gradle

dependencies {
  compile "org.apache.geode:geode-core:$VERSION"
}

Main Concepts and Components

Caches are an abstraction that describe a node in an Apache Geode distributed system.

Within each cache, you define data regions. Data regions are analogous to tables in a relational database and manage data in a distributed fashion as name/value pairs. A replicated region stores identical copies of the data on each cache member of a distributed system. A partitioned region spreads the data among cache members. After the system is configured, client applications can access the distributed data in regions without knowledge of the underlying system architecture. You can define listeners to receive notifications when data has changed, and you can define expiration criteria to delete obsolete data in a region.

Locators provide clients with both discovery and server load balancing services. Clients are configured with locator information, and the locators maintain a dynamic list of member servers. The locators provide clients with connection information to a server.

Apache Geode includes the following features:

  • Combines redundancy, replication, and a “shared nothing” persistence architecture to deliver fail-safe reliability and performance.
  • Horizontally scalable to thousands of cache members, with multiple cache topologies to meet different enterprise needs. The cache can be distributed across multiple computers.
  • Asynchronous and synchronous cache update propagation.
  • Delta propagation distributes only the difference between old and new versions of an object (delta) instead of the entire object, resulting in significant distribution cost savings.
  • Reliable asynchronous event notifications and guaranteed message delivery through optimized, low latency distribution layer.
  • Data awareness and real-time business intelligence. If data changes as you retrieve it, you see the changes immediately.
  • Integration with Spring Framework to speed and simplify the development of scalable, transactional enterprise applications.
  • JTA compliant transaction support.
  • Cluster-wide configurations that can be persisted and exported to other clusters.
  • Remote cluster management through HTTP.
  • REST APIs for REST-enabled application development.
  • Rolling upgrades may be possible, but they will be subject to any limitations imposed by new features.

Building this Release from Source

See BUILDING.md for instructions on how to build the project.

Running Tests

See TESTING.md for instructions on how to run tests.

Geode in 5 minutes

Geode requires installation of JDK version 1.8. After installing Apache Geode, start a locator and server:

$ gfsh
gfsh> start locator
gfsh> start server

Create a region:

gfsh> create region --name=hello --type=REPLICATE

Write a client application (this example uses a Gradle build script):

build.gradle

apply plugin: 'java'
apply plugin: 'application'

mainClassName = 'HelloWorld'

repositories { mavenCentral() }
dependencies {
  compile 'org.apache.geode:geode-core:1.4.0'
  runtime 'org.slf4j:slf4j-log4j12:1.7.24'
}

src/main/java/HelloWorld.java

import java.util.Map;
import org.apache.geode.cache.Region;
import org.apache.geode.cache.client.*;

public class HelloWorld {
  public static void main(String[] args) throws Exception {
    ClientCache cache = new ClientCacheFactory()
      .addPoolLocator("localhost", 10334)
      .create();
    Region<String, String> region = cache
      .<String, String>createClientRegionFactory(ClientRegionShortcut.CACHING_PROXY)
      .create("hello");

    region.put("1", "Hello");
    region.put("2", "World");

    for (Map.Entry<String, String>  entry : region.entrySet()) {
      System.out.format("key = %s, value = %s\n", entry.getKey(), entry.getValue());
    }
    cache.close();
  }
}

Build and run the HelloWorld example:

$ gradle run

The application will connect to the running cluster, create a local cache, put some data in the cache, and print the cached data to the console:

key = 1, value = Hello
key = 2, value = World

Finally, shutdown the Geode server and locator:

gfsh> shutdown --include-locators=true

For more information see the Geode Examples repository or the documentation.

Application Development

Apache Geode applications can be written in these client technologies:

The following libraries are available external to the Apache Geode project:

Export Control

This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See http://www.wassenaar.org/ for more information.

The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code.

The following provides more details on the included cryptographic software: