commit	ba0389f57cf9d1321e3762f21da04b62d04559d5	[log] [tgz]
author	Randall Hauch <rhauch@gmail.com>	Thu Feb 22 09:39:59 2018 +0000
committer	Damian Guy <damian.guy@gmail.com>	Thu Feb 22 09:42:40 2018 +0000
tree	a695758b1c3b4b4fdd40abcc726898b212a9e152
parent	b3a93073ccb66a711644a62d097619121005b511 [diff]

KAFKA-6577: Fix Connect system tests and add debug messages

**NOTE: This should be backported to the `1.1` branch, and is currently a blocker for 1.1.**

The `connect_test.py::ConnectStandaloneFileTest.test_file_source_and_sink` system test is failing with the SASL configuration without a sufficient explanation. During the test, the Connect worker fails to start, but the Connect log contains no useful information. There are actual several things compounding to cause the failure and make it difficult to understand the problem.

First, the `tests/kafkatest/tests/connect/templates/connect_standalone.properties` is only adding in the broker's security configuration with the `producer.` and `consumer.` prefixes, but is not adding them with no prefix. The worker uses the AdminClient to connect to the broker to get the Kafka cluster ID and to manage the three internal topics, and the AdminClient is configured via top-level properties. Because the SASL test requires the clients all connect using SASL, the lack of broker security configs means the AdminClient was attempting and failing to connect to the broker. This is corrected by adding the broker's security configuration to the Connect worker configuration file at the top-level. (This was already being done in the `connect_distributed.properties` file.)

Second, the default `request.timeout.ms` for the AdminClient (and the other clients) is 120 seconds, so the AdminClient was retrying for 120 seconds before it would give up and thrown an error. However, the test was only waiting for 60 seconds before determining that the service failed to start. This can be corrected by setting `request.timeout.ms=10000` in the Connect distributed and standalone worker configurations.

Third, the Connect workers were recently changed to lookup the Kafka cluster ID before it started the herder. This is unlike the older uses of the AdminClient to find and manage the internal topics, where failure to connect was not necessarily logged correctly but nevertheless still skipped over, relying upon broker auto-topic creation to create the internal topics. (This may be why the test did not fail prior to the recent change to always require a successful AdminClient connection.) Although the worker never got this far in its startup process, the fact that we missed such an error since the prior releases means that failure to connect with the AdminClient was not being properly reported.

The `ConnectStandaloneFileTest.test_file_source_and_sink` system tests were run locally prior to this fix, and they failed as with the nightlies. Once these fixes were made, the locally run system tests passed.

Author: Randall Hauch <rhauch@gmail.com>

Reviewers: Konstantine Karantasis <konstantine@confluent.io>, Ewen Cheslack-Postava <me@ewencp.org>

Closes #4610 from rhauch/kafka-6577-trunk

(cherry picked from commit fc19c3e6f243a8d1b3e27cdc912dc092bbd342e0)
Signed-off-by: Damian Guy <damian.guy@gmail.com>

9 files changed

tree: a695758b1c3b4b4fdd40abcc726898b212a9e152

README.md

Apache Kafka

See our web site for details on the project.

You need to have Gradle and Java installed.

Kafka requires Gradle 3.0 or higher.

Java 7 should be used for building in order to support both Java 7 and Java 8 at runtime.

First bootstrap and download the wrapper

cd kafka_source_dir
gradle

Now everything else will work.

Build a jar and run it

./gradlew jar

Follow instructions in http://kafka.apache.org/documentation.html#quickstart

Build source jar

./gradlew srcJar

Build aggregated javadoc

./gradlew aggregatedJavadoc

Build javadoc and scaladoc

./gradlew javadoc
./gradlew javadocJar # builds a javadoc jar for each module
./gradlew scaladoc
./gradlew scaladocJar # builds a scaladoc jar for each module
./gradlew docsJar # builds both (if applicable) javadoc and scaladoc jars for each module

Run unit/integration tests

./gradlew test # runs both unit and integration tests
./gradlew unitTest
./gradlew integrationTest

Force re-running tests without code change

./gradlew cleanTest test
./gradlew cleanTest unitTest
./gradlew cleanTest integrationTest

Running a particular unit/integration test

./gradlew -Dtest.single=RequestResponseSerializationTest core:test

Running a particular test method within a unit/integration test

./gradlew core:test --tests kafka.api.ProducerFailureHandlingTest.testCannotSendToInternalTopic
./gradlew clients:test --tests org.apache.kafka.clients.MetadataTest.testMetadataUpdateWaitTime

Running a particular unit/integration test with log4j output

Change the log4j setting in either clients/src/test/resources/log4j.properties or core/src/test/resources/log4j.properties

./gradlew -i -Dtest.single=RequestResponseSerializationTest core:test

Generating test coverage reports

Generate coverage reports for the whole project:

./gradlew reportCoverage

Generate coverage for a single module, i.e.:

./gradlew clients:reportCoverage

Building a binary release gzipped tar ball

./gradlew clean
./gradlew releaseTarGz

The above command will fail if you haven't set up the signing key. To bypass signing the artifact, you can run:

./gradlew releaseTarGz -x signArchives

The release file can be found inside ./core/build/distributions/.

Cleaning the build

./gradlew clean

Running a task on a particular version of Scala (either 2.11.x or 2.12.x)

Note that if building the jars with a version other than 2.11.12, you need to set the SCALA_VERSION variable or change it in bin/kafka-run-class.sh to run the quick start.

You can pass either the major version (eg 2.11) or the full version (eg 2.11.12):

./gradlew -PscalaVersion=2.11 jar
./gradlew -PscalaVersion=2.11 test
./gradlew -PscalaVersion=2.11 releaseTarGz

Scala 2.12.x requires Java 8.

Running a task for a specific project

This is for core, examples and clients

./gradlew core:jar
./gradlew core:test

Listing all gradle tasks

./gradlew tasks

Building IDE project

Note that this is not strictly necessary (IntelliJ IDEA has good built-in support for Gradle projects, for example).

./gradlew eclipse
./gradlew idea

The eclipse task has been configured to use ${project_dir}/build_eclipse as Eclipse‘s build directory. Eclipse’s default build directory (${project_dir}/bin) clashes with Kafka‘s scripts directory and we don’t use Gradle's build directory to avoid known issues with this configuration.

Building the jar for all scala versions and for all projects

./gradlew jarAll

Running unit/integration tests for all scala versions and for all projects

./gradlew testAll

Building a binary release gzipped tar ball for all scala versions

./gradlew releaseTarGzAll

Publishing the jar for all version of Scala and for all projects to maven

./gradlew uploadArchivesAll

Please note for this to work you should create/update ${GRADLE_USER_HOME}/gradle.properties (typically, ~/.gradle/gradle.properties) and assign the following variables

mavenUrl=
mavenUsername=
mavenPassword=
signing.keyId=
signing.password=
signing.secretKeyRingFile=

Publishing the streams quickstart archetype artifact to maven

For the Streams archetype project, one cannot use gradle to upload to maven; instead the mvn deploy command needs to be called at the quickstart folder:

cd streams/quickstart
mvn deploy

Please note for this to work you should create/update user maven settings (typically, ${USER_HOME}/.m2/settings.xml) to assign the following variables

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                       https://maven.apache.org/xsd/settings-1.0.0.xsd">
...                           
<servers>
   ...
   <server>
      <id>apache.snapshots.https</id>
      <username>${maven_username}</username>
      <password>${maven_password}</password>
   </server>
   <server>
      <id>apache.releases.https</id>
      <username>${maven_username}</username>
      <password>${maven_password}</password>
    </server>
    ...
 </servers>
 ...

Installing the jars to the local Maven repository

./gradlew installAll

Building the test jar

./gradlew testJar

Determining how transitive dependencies are added

./gradlew core:dependencies --configuration runtime

Determining if any dependencies could be updated

./gradlew dependencyUpdates

Running code quality checks

There are two code quality analysis tools that we regularly run, findbugs and checkstyle.

Checkstyle

Checkstyle enforces a consistent coding style in Kafka. You can run checkstyle using:

./gradlew checkstyleMain checkstyleTest

The checkstyle warnings will be found in reports/checkstyle/reports/main.html and reports/checkstyle/reports/test.html files in the subproject build directories. They are also are printed to the console. The build will fail if Checkstyle fails.

Findbugs

Findbugs uses static analysis to look for bugs in the code. You can run findbugs using:

./gradlew findbugsMain findbugsTest -x test

The findbugs warnings will be found in reports/findbugs/main.html and reports/findbugs/test.html files in the subproject build directories. Use -PxmlFindBugsReport=true to generate an XML report instead of an HTML one.

Common build options

The following options should be set with a -P switch, for example ./gradlew -PmaxParallelForks=1 test.

commitId: sets the build commit ID as .git/HEAD might not be correct if there are local commits added for build purposes.
mavenUrl: sets the URL of the maven deployment repository (file://path/to/repo can be used to point to a local repository).
maxParallelForks: limits the maximum number of processes for each task.
showStandardStreams: shows standard out and standard error of the test JVM(s) on the console.
skipSigning: skips signing of artifacts.
testLoggingEvents: unit test events to be logged, separated by comma. For example ./gradlew -PtestLoggingEvents=started,passed,skipped,failed test.
xmlFindBugsReport: enable XML reports for findBugs. This also disables HTML reports as only one can be enabled at a time.

Running in Vagrant

See vagrant/README.md.

Contribution

Apache Kafka is interested in building the community; we would welcome any thoughts or patches. You can reach us on the Apache mailing lists.

To contribute follow the instructions here:

http://kafka.apache.org/contributing.html