/tls/certs/truststore.jks\n-Djavax.net.ssl.keyStorePassword=test1234\n-Djavax.net.ssl.trustStorePassword=test1234\n```\n\n### Important configuration files\n\n kafka server.properties
\n\nlisteners=SSL://kafka.confluent.local:9093\nadvertised.listeners=SSL://kafka.confluent.local:9093\nsecurity.inter.broker.protocol=SSL\nssl.truststore.location=/var/lib/secret/truststore.jks\nssl.truststore.password=test1234\nssl.keystore.location=/var/lib/secret/server.keystore.jks\nssl.keystore.password=test1234\nssl.client.auth=required\n# To use TLS based authorization\nauthorizer.class.name=kafka.security.auth.SimpleAclAuthorizer\nsuper.users=User:CN=kafka.confluent.local,L=London,O=Confluent,C=UK\n
\n \n\nkafka consumer and producer configuration
\n\nbootstrap.servers=kafka.conflent.local:9093\nsecurity.protocol=SSL\nssl.truststore.location=/var/lib/secret/truststore.jks\nssl.truststore.password=test1234\nssl.keystore.location=/var/lib/secret/client.keystore.jks\nssl.keystore.password=test1234\nssl.key.password=test1234\n
\n \n\n#### For further information\n* [kafka documentation on TLS](http://kafka.apache.org/documentation.html#security_ssl)\n* [Confluent documentation on TLS authentication](https://docs.confluent.io/current/kafka/authentication_ssl.html)\n* [Confluent documentation on TLS key generation](https://docs.confluent.io/current/tutorials/security_tutorial.html#generating-keys-certs)\n\n## Kerberos (GSSAPI) authentication without TLS\nThis example contains a basic KDC server and configure both zookeeper and kafka with Kerberos authentication and authorization. Credentials are created without password, a keytab containing credentials is available in a Docker volume named \"secret\". The following credential are automatically created in the KDC database:\n1. __kafka/admin__ - to access zookeeper\n2. __kafka_producer/producer__ - to access kafka as a producer\n3. __kafka_consumer/consumer__ - to access kafka as a consumer\n\n### Usage\n```bash\ncd kerberos\n# Scripts orchestrating the docker-compose services\n./up\n# Using kinit with a keytab for authentication then invoking kafka interfaces\ndocker-compose exec kafka bash -c 'kinit -k -t /var/lib/secret/kafka.key kafka_producer/producer && kafka-console-producer --broker-list kafka:9093 --topic test --producer.config /etc/kafka/consumer.properties'\ndocker-compose exec kafka bash -c 'kinit -k -t /var/lib/secret/kafka.key kafka_consumer/consumer && kafka-console-consumer --bootstrap-server kafka:9093 --topic test --consumer.config /etc/kafka/consumer.properties --from-beginning'\n```\n\n### Important configuration files\n\nzookeeper properties
\n\nauthProvider.1 = org.apache.zookeeper.server.auth.SASLAuthenticationProvider\nrequireClientAuthScheme=sasl\n
\n \n\nzookeeper server and client jaas configuration
\n\nServer {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n\t\tuseTicketCache=false\n keyTab=\"/var/lib/secret/kafka.key\"\n principal=\"zookeeper/zookeeper.kerberos_default@TEST.CONFLUENT.IO\";\n};\n\n \n\nkafka server.properties
\n\nlisteners=SASL_PLAINTEXT://kafka:9093\nadvertised.listeners=SASL_PLAINTEXT://kafka:9093\nsecurity.inter.broker.protocol=SASL_PLAINTEXT\nsasl.enabled.mechanisms=GSSAPI\nsasl.mechanism.inter.broker.protocol=GSSAPI\nsecurity.inter.broker.protocol=SASL_PLAINTEXT\nsasl.kerberos.service.name=kafka\nallow.everyone.if.no.acl.found=false\nsuper.users=User:admin;User:kafka\nauthorizer.class.name=kafka.security.auth.SimpleAclAuthorizer\n
\n \n\nkafka server and client jaas configuration
\n\n/*\n * Cluster kerberos services\n */\nKafkaServer {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/var/lib/secret/kafka.key\"\n principal=\"kafka/kafka.kerberos_default@TEST.CONFLUENT.IO\";\n};\n\n/*\n * For client and broker identificatoin\n */\nKafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/var/lib/secret/kafka.key\"\n principal=\"admin/kafka.kerberos_default@TEST.CONFLUENT.IO\";\n};\n\n/*\n * For Zookeeper authentication\n */\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n\t\tuseTicketCache=false\n keyTab=\"/var/lib/secret/kafka.key\"\n principal=\"kafka/kafka.kerberos_default@TEST.CONFLUENT.IO\";\n};\n\n \n\n\tkafka consumer and producer configuration
\n\nbootstrap.servers=kafka:9093\nsecurity.protocol=SASL_PLAINTEXT\nsasl.kerberos.service.name=kafka\nsasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \\\n\t\t\t\t\t\t\t\t useTicketCache=true\n
\n \n\n\n#### For further information\n* [Confluent documentation on GSSAPI authentication](https://docs.confluent.io/current/kafka/authentication_sasl_gssapi.html)\n* [Confluent documentation on ACL](https://docs.confluent.io/current/kafka/authorization.html)\n\n## Oauth authentication via TLS encryption\n\nKafka supports SASL authentication via Oauth bearer tokens. A sample playbook for secured oauth token authentication is contained in the oauth subfolder of this repository.\n\n### Usage\n\nPrerequisites: jdk8, maven, docker-compose, openssl.\n\n```bash\ncd oauth\n./up\n```\n\nIn this sample playbook both the identity of brokers (`sasl.mechanism.inter.broker.protocol=OAUTHBEARER` within server.properties) and the identity of clients (`sasl.mechanism=OAUTHBEARER` within consumer.properties) are verified by the brokers using oauth bearer tokens.\n\nWithin this sample playbook oauth bearer tokens are generated and validated using the `jjwt` library without communication to an authorization server. In real life, this would be different.\n\nThe class `OauthBearerLoginCallbackHandler` is used by the clients and by brokers to generate a JWT token using a shared secret. This class is configured within the `client.properties` file:\n\nNote that the client does not need to have a keystore configured, since client authentication is achieved using bearer tokens.\nStill it needs a truststore to store the brokers certificate authorities.\n\n\n\tkafka consumer and producer configuration
\n\nsecurity.protocol=SASL_SSL\nsasl.mechanism=OAUTHBEARER\nsasl.login.callback.handler.class=io.confluent.examples.authentication.oauth.OauthBearerLoginCallbackHandler\nssl.truststore.location=/etc/kafka/kafka.client.truststore.jks\nssl.truststore.password=secret\n
\n \n\nThe `OauthBearerLoginCallbackHandler` class is also configured for broker clients within the `server.properties` file (see below). The `server.properties` file must also include a reference to the token validator class (`OauthBearerValidatorCallbackHandler`):\n\n\n\tkafka broker configuration
\n\nlisteners=SASL_SSL://kafka.confluent.local:9093\nadvertised.listeners=SASL_SSL://kafka.confluent.local:9093\nsecurity.inter.broker.protocol=SASL_SSL\nsasl.mechanism.inter.broker.protocol=OAUTHBEARER\nsasl.enabled.mechanisms=OAUTHBEARER\nlistener.name.sasl_ssl.oauthbearer.sasl.server.callback.handler.class=io.confluent.examples.authentication.oauth.OauthBearerValidatorCallbackHandler\nlistener.name.sasl_ssl.oauthbearer.sasl.login.callback.handler.class=io.confluent.examples.authentication.oauth.OauthBearerLoginCallbackHandler\nssl.truststore.location=/etc/kafka/kafka.server.truststore.jks\nssl.truststore.password=secret\nssl.keystore.location=/etc/kafka/kafka.server.keystore.jks\nssl.keystore.password=secret\nssl.key.password=secret\n
\n \n\nKafka brokers need a keystore to store its private certificate as well as a truststore to verify the identity of other brokers.\n\n### Further information\n\n* [Confluent documentation on Oauth authentication](https://docs.confluent.io/current/kafka/authentication_sasl/authentication_sasl_oauth.html)\n* [Blog Post](https://medium.com/@jairsjunior/how-to-setup-oauth2-mechanism-to-a-kafka-broker-e42e72839fe)\n\n## Schema registry basic security\n\nAccording to documentation the schema registry plugin only supports SSL principals, but there is an undocumented separate authentication possibility via Jetty Authentication.\n\n```bash\ncd schema-registry-basic-auth\n./up\n```\n\nNow you can access the schema registry REST interface on `http://localhost:8089`\n\nNote that in order to test the schema registry properly, you need to either `curl` into it, or use the `kafka-avro-consule-producer` and consumer. The latter require special considerations.\n\nFirst, access via `curl`:\n\n```\ncurl -X GET http://localhost:8089 -u admin:admin\n```\n\nIf you want to try out the console producer, you need to exec into the schema-registry docker image and then run the producer:\n\n```\ndocker-compose exec schema-registry bash\nkafka-avro-console-producer --broker-list kafka:9092 --topic avro-test --property \\\n value.schema='{\"type\":\"record\",\"name\":\"myrecord\",\"fields\":[{\"name\":\"f1\",\"type\":\"string\"}]}' \\\n --property basic.auth.credentials.source=USER_INFO \\\n --property schema.registry.basic.auth.user.info=write:write\n\n> {\"f1\": \"value1\"}\n> {\"f1\": \"value2\"}\n> ^D\n```\n\nNote that the official documentation is wrong on two accounts. First, to define the source, you need to use `basic.auth.credentials.source` without the `schema.registry` in front of it.\n\nSecond, user authentication via a property file gets ignored, you need to pass the credentials via `--property`.\n\n## Schema registry semi-open security\n\nThis playbook is an example of configuration where Schema Registry is configured for accepting request on `http` and `https`.\nRequests on the `http` endpoint are actually identified as the `ANONYMOUS` user. This is possible thanks to the `confluent.schema.registry.anonymous.principal=true` option.\n\nThe following ACLs are configured:\n- `sr-acl-cli --config /etc/schema-registry/schema-registry.properties --add -s '*' -p 'ANONYMOUS' -o 'SUBJECT_READ'`\n- `sr-acl-cli --config /etc/schema-registry/schema-registry.properties --add -p 'ANONYMOUS' -o 'GLOBAL_SUBJECTS_READ'`\n- `sr-acl-cli --config /etc/schema-registry/schema-registry.properties --add -p 'ANONYMOUS' -o 'GLOBAL_COMPATIBILITY_READ'`\n- `sr-acl-cli --config /etc/schema-registry/schema-registry.properties --add -s '*' -p 'C=UK,O=Confluent,L=London,CN=schema-registry' -o '*'`\n\nWith this configuration, ` curl -X GET http://localhost:8089/subjects/` is successful, but the `ANONYMOUS` user does not have the privileges to write new schemas.\nOnly the client with the TLS client certificate `C=UK,O=Confluent,L=London,CN=schema-registry` can write new schemas, this could be for example your CI tool or an admin user.\n"
},
{
"path": "TlsCheatsheet.md",
"content": "# TLS Cheat Sheet\n\n## Introduction \n\nThis cheat sheet contains common commands regarding TLS certificate generation and TLS troubleshooting. If you are looking for a script to generate keystore , certificate authority and certificates, I recommend you to check out [confluent kafka-generate-ssl.sh script](https://github.com/confluentinc/confluent-platform-security-tools/blob/master/kafka-generate-ssl.sh)\n\n## Generating self-signed certificates or a new Certificate Authority\n\n```bash\nopenssl req -new -nodes -x509 -days 3650 -newkey rsa:2048 -keyout sever.key -out certs/server.crt -config $CONFIG_PATH\n```\n\n\n## Generating certificate signing request\n\n```bash\nopenssl req -new -newkey rsa:2048 -keyout server.key -out server.csr -config $CONFIG_PATH -nodes\n```\n\n## Displaying content of a signing request\n\n```bash\nopenssl req -text -in $CERT\n```\n\n## Displaying content of a certificate that a server presents\n\n```bash\nopenssl s_client -showcerts -connect www.example.com:443\n```\n\n## Verifying that server certificate was signed by a CA\n\n```bash\ncurl --cacert /var/lib/secret/ca.pem https://www.example:8443\n```\n\n## Signing certificate signing request\n\n```bash\nopenssl x509 -req -days $DURATION -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -extfile $CONFIG_PATH\n```\n\n## Generate a signed certificate with keytool\n\n```bash\n# First create the keystore \nkeytool -keystore keystore.server.jks -alias server -validity $DURATION -genkey -keyalg RSA\n\n# Generate a certificate signing request and export it to a file\nkeytool -keystore keystore.server.jks -alias server -certreq -file $KEYSTORE_SIGN_REQUEST\n\n# Sign the certificate request with OpenSSL and a CA\nopenssl x509 -req -CA ca.crt -CAkey ca.key -in $KEYSTORE_SIGN_REQUEST -out $KEYSTORE_SIGNED_CERT -days $VALIDITY_IN_DAYS -CAcreateserial\n\n# Importing the signed certificate to the keystore\nkeytool -keystore $KEYSTORE_WORKING_DIRECTORY/$KEYSTORE_FILENAME -alias localhost -import -file $KEYSTORE_SIGNED_CERT\n```\n\n## Displaying content of a signed certificate\n\n```bash\nopenssl x509 -text -in $CERT\n```\n\n## Importing signed certificate with its private key into a keystore\n\n```bash\n# Exporting certificate to PKCS12 format\nopenssl pkcs12 -export -in server.crt -inkey server.key -chain -CAfile ca.pem -name \"kafka.confluent.local\" -out server.p12 -password pass:$PASSWORD\n\n# Importing PKCS12 into another keystore (or create it)\nkeytool -importkeystore -deststorepass $PASSWORD -destkeystore server.keystore.jks -srckeystore server.p12 -deststoretype PKCS12 -srcstoretype PKCS12 -noprompt -srcstorepass $PASSWORD\n```\n\n## Import certificate into a keystore\n\n```bash\nkeytool -keystore truststore.jks -alias $ALIAS -import -file $CRT_FILE -storepass $PASSWORD -noprompt -storetype PKCS12 \n```\n\n\n## Example of OpenSSL configuration file to generate a CA\n\n```\n[ policy_match ]\ncountryName = match\nstateOrProvinceName = match\norganizationName = match\norganizationalUnitName = optional\ncommonName = supplied\nemailAddress = optional\n\n[ req ]\nprompt = no\ndistinguished_name = dn\ndefault_md = sha256\ndefault_bits = 4096\nx509_extensions = v3_ca\n\n[ dn ]\ncountryName = UK\norganizationName = Confluent\nlocalityName = London\ncommonName = kafka.confluent.local\n\n[ v3_ca ]\nsubjectKeyIdentifier=hash\nbasicConstraints = critical,CA:true\nauthorityKeyIdentifier=keyid:always,issuer:always\nkeyUsage = critical,keyCertSign,cRLSign\n```\n\n## Example of OpenSSL configuration file to generate a server certificate\n\n```\n[req]\nprompt = no\ndistinguished_name = dn\ndefault_md = sha256\ndefault_bits = 4096\nreq_extensions = v3_req\n\n[ dn ]\ncountryName = UK\norganizationName = Confluent\nlocalityName = London\ncommonName=kafka.confluent.local\n\n[ v3_req ]\nsubjectKeyIdentifier = hash\nbasicConstraints = CA:FALSE\nnsComment = \"OpenSSL Generated Certificate\"\nkeyUsage = critical, digitalSignature, keyEncipherment\nextendedKeyUsage = serverAuth, clientAuth\nsubjectAltName = @alt_names\n\n[ alt_names ]\nDNS.1=kafka.confluent.local\n```\n\n## Example of OpenSSL configuration file to generate a client certificate\n\n```\n[req]\nprompt = no\ndistinguished_name = dn\ndefault_md = sha256\ndefault_bits = 4096\nreq_extensions = v3_req\n\n[ dn ]\ncountryName = UK\norganizationName = Confluent\nlocalityName = London\ncommonName=kafka.confluent.local\n\n[ v3_req ]\nsubjectKeyIdentifier = hash\nbasicConstraints = CA:FALSE\nnsComment = \"OpenSSL Generated Certificate\"\nkeyUsage = critical, digitalSignature, keyEncipherment\nextendedKeyUsage = clientAuth\n```\n"
},
{
"path": "acls/docker-compose.yaml",
"content": "---\nversion: '3'\nservices:\n zookeeper:\n image: confluentinc/cp-zookeeper:5.4.0\n hostname: zookeeper\n container_name: zookeeper\n environment:\n ZOOKEEPER_SERVER_ID: 1\n ZOOKEEPER_CLIENT_PORT: 2182\n ZOOKEEPER_TICK_TIME: \"2000\"\n KAFKA_JMX_PORT: 9999\n KAFKA_JMX_HOSTNAME: localhost\n KAFKA_OPTS: -Djava.security.auth.login.config=/tmp/zookeeper.sasl.jaas.conf \n -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider\n ports:\n - 2182:2182\n volumes:\n - \"$PWD/zookeeper.sasl.jaas.conf:/tmp/zookeeper.sasl.jaas.conf\" \n\n kafka-1:\n build: kafka/\n hostname: kafka-1\n container_name: kafka-1\n depends_on:\n - zookeeper\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2182\n KAFKA_LISTENERS: INTERNAL://kafka-1:19093, EXTERNAL://0.0.0.0:9093\n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT\n KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-1:19093, EXTERNAL://localhost:9093\n KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL\n KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter\n KAFKA_CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: kafka-1:19093\n KAFKA_CONFLUENT_METRICS_REPORTER_SASL_MECHANISM: \"SCRAM-SHA-256\"\n KAFKA_CONFLUENT_METRICS_REPORTER_SECURITY_PROTOCOL: SASL_PLAINTEXT\n KAFKA_CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1\n KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1\n KAFKA_SASL_ENABLED_MECHANISMS: \"SCRAM-SHA-256\"\n KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: \"SCRAM-SHA-256\"\n KAFKA_ZOOKEEPER_SET_ACL: \"true\"\n KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.auth.SimpleAclAuthorizer\n KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: \"false\"\n KAFKA_SUPER_USERS: \"User:kafka;User:admin\"\n KAFKA_JMX_PORT: 9999\n KAFKA_JMX_HOSTNAME: kafka-1\n KAFKA_OPTS: \"-Djava.security.auth.login.config=/tmp/kafka.sasl.jaas.conf\"\n ports:\n - 9093:9093\n\n kafka-2:\n build: kafka/\n hostname: kafka-2\n container_name: kafka-2\n depends_on:\n - zookeeper\n environment:\n KAFKA_BROKER_ID: 2\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2182\n KAFKA_LISTENERS: INTERNAL://kafka-2:19094, EXTERNAL://0.0.0.0:9094\n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT\n KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-2:19094, EXTERNAL://localhost:9094\n KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL\n KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter\n KAFKA_CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: kafka-2:19094\n KAFKA_CONFLUENT_METRICS_REPORTER_SASL_MECHANISM: \"SCRAM-SHA-256\"\n KAFKA_CONFLUENT_METRICS_REPORTER_SECURITY_PROTOCOL: SASL_PLAINTEXT\n KAFKA_CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1\n KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1\n KAFKA_SASL_ENABLED_MECHANISMS: \"SCRAM-SHA-256\"\n KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: \"SCRAM-SHA-256\"\n KAFKA_ZOOKEEPER_SET_ACL: \"true\"\n KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.auth.SimpleAclAuthorizer\n KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: \"false\"\n KAFKA_SUPER_USERS: \"User:kafka;User:admin\"\n KAFKA_JMX_PORT: 9999\n KAFKA_JMX_HOSTNAME: kafka-2\n KAFKA_OPTS: \"-Djava.security.auth.login.config=/tmp/kafka.sasl.jaas.conf\"\n ports:\n - 9094:9094\n"
},
{
"path": "acls/kafka/Dockerfile",
"content": "FROM confluentinc/cp-enterprise-kafka:5.4.0\n\nMAINTAINER sven@confluent.io\n\n# Make sure the log directory is world-writable\nRUN echo \"===> Creating authorizer logs dir ...\" \\\n && mkdir -p /var/log/kafka-auth-logs \\\n && chmod -R ag+w /var/log/kafka-auth-logs\n\nCOPY log4j.properties.template /etc/confluent/docker/log4j.properties.template\n\nCOPY *.conf /tmp/\n\n"
},
{
"path": "acls/kafka/admin.conf",
"content": "sasl.mechanism=SCRAM-SHA-256\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"admin\" \\\n password=\"admin-pass\";\n"
},
{
"path": "acls/kafka/consumer.conf",
"content": "sasl.mechanism=SCRAM-SHA-256\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"consumer\" \\\n password=\"consumer-pass\";\n"
},
{
"path": "acls/kafka/kafka.conf",
"content": "sasl.mechanism=SCRAM-SHA-256\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"kafka\" \\\n password=\"kafka-pass\";\n"
},
{
"path": "acls/kafka/kafka.sasl.jaas.conf",
"content": "KafkaServer {\n org.apache.kafka.common.security.scram.ScramLoginModule required\n username=\"kafka\"\n password=\"kafka-pass\";\n};\nKafkaClient {\n org.apache.kafka.common.security.scram.ScramLoginModule required\n username=\"kafka\"\n password=\"kafka-pass\";\n};\nClient {\n org.apache.zookeeper.server.auth.DigestLoginModule required\n username=\"admin\"\n password=\"password\";\n};\n\n"
},
{
"path": "acls/kafka/kafkacat.conf",
"content": "security.protocol=SASL_PLAINTEXT\nsasl.mechanisms=SCRAM-SHA-256\nsasl.username=kafka\nsasl.password=kafka-pass\n"
},
{
"path": "acls/kafka/log4j.properties.template",
"content": "log4j.rootLogger={{ env[\"KAFKA_LOG4J_ROOT_LOGLEVEL\"] | default('INFO') }}, stdout\n\nlog4j.appender.stdout=org.apache.log4j.ConsoleAppender\nlog4j.appender.stdout.layout=org.apache.log4j.PatternLayout\nlog4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.appender.authorizerAppender=org.apache.log4j.DailyRollingFileAppender\nlog4j.appender.authorizerAppender.DatePattern='.'yyyy-MM-dd-HH\nlog4j.appender.authorizerAppender.File=/var/log/kafka-auth-logs/kafka-authorizer.log\nlog4j.appender.authorizerAppender.layout=org.apache.log4j.PatternLayout\nlog4j.appender.authorizerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.additivity.kafka.authorizer.logger=false\n\n{% set loggers = {\n 'kafka': 'INFO',\n 'kafka.network.RequestChannel$': 'WARN',\n 'kafka.producer.async.DefaultEventHandler': 'DEBUG',\n 'kafka.request.logger': 'WARN',\n 'kafka.controller': 'TRACE',\n 'kafka.log.LogCleaner': 'INFO',\n 'state.change.logger': 'TRACE',\n 'kafka.authorizer.logger': 'DEBUG, authorizerAppender'\n } -%}\n\n\n{% if env['KAFKA_LOG4J_LOGGERS'] %}\n{% set loggers = parse_log4j_loggers(env['KAFKA_LOG4J_LOGGERS'], loggers) %}\n{% endif %}\n\n{% for logger,loglevel in loggers.iteritems() %}\nlog4j.logger.{{logger}}={{loglevel}}\n{% endfor %}\n\n"
},
{
"path": "acls/kafka/producer.conf",
"content": "sasl.mechanism=SCRAM-SHA-256\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"producer\" \\\n password=\"producer-pass\";\n"
},
{
"path": "acls/up",
"content": "#!/bin/sh\n \ndocker-compose up -d --build\n\n# Creating the user kafka\n# kafka is configured as a super user, no need for additional ACL\ndocker-compose exec kafka-1 kafka-configs --zookeeper zookeeper:2182 --alter --add-config 'SCRAM-SHA-256=[password=kafka-pass],SCRAM-SHA-512=[password=kafka-pass]' --entity-type users --entity-name kafka\ndocker-compose exec kafka-1 kafka-configs --zookeeper zookeeper:2182 --alter --add-config 'SCRAM-SHA-256=[password=admin-pass],SCRAM-SHA-512=[password=admin-pass]' --entity-type users --entity-name admin\ndocker-compose exec kafka-1 kafka-configs --zookeeper zookeeper:2182 --alter --add-config 'SCRAM-SHA-256=[password=producer-pass],SCRAM-SHA-512=[password=producer-pass]' --entity-type users --entity-name producer\ndocker-compose exec kafka-1 kafka-configs --zookeeper zookeeper:2182 --alter --add-config 'SCRAM-SHA-256=[password=consumer-pass],SCRAM-SHA-512=[password=consumer-pass]' --entity-type users --entity-name consumer\n\n# ACLs\ndocker-compose exec kafka-1 kafka-acls --authorizer-properties zookeeper.connect=zookeeper:2182 --add --allow-principal User:producer --producer --topic=*\ndocker-compose exec kafka-1 kafka-acls --authorizer-properties zookeeper.connect=zookeeper:2182 --add --allow-principal User:consumer --consumer --topic=* --group=*\n\necho \"Example configuration:\"\necho \"-> kafka-console-producer --broker-list localhost:9093 --producer.config kafka/producer.conf --topic test\"\necho \"-> kafka-console-consumer --bootstrap-server localhost:9094 --consumer.config kafka/consumer.conf --topic test --from-beginning\"\necho \"ZooKeeper shell with authorization from host:\"\necho \"-> KAFKA_OPTS=\\\"-Djava.security.auth.login.config=zookeeper.sasl.jaas.conf\\\" zookeeper-shell localhost:2182\"\necho \"ZooKeeper shell with authorization within container (KAFKA_OPTS already set):\"\necho \"-> docker-compose exec kafka-1 zookeeper-shell zookeeper:2182\"\necho \"Kafkacat with authorization from host:\"\necho \"-> kafkacat -L -b localhost:9094 -F kafka/kafkacat.conf\"\n"
},
{
"path": "acls/zookeeper.sasl.jaas.conf",
"content": "Server {\n org.apache.zookeeper.server.auth.DigestLoginModule required\n user_admin=\"password\";\n};\nClient {\n org.apache.zookeeper.server.auth.DigestLoginModule required\n username=\"admin\"\n password=\"password\";\n};\n"
},
{
"path": "apache-kafka-with-zk3.5-and-tls/.gitignore",
"content": "bin/\ncerts/\ncerts-old/\ntmp-dir\nimages/\nzookeeper.properties\n"
},
{
"path": "apache-kafka-with-zk3.5-and-tls/README.md",
"content": "# Apache Kafka 2.4 (trunk) with Zookeeper 3.5.5\n\nThis playbook show the current (as of August 2019) necessary steps to enable a secured TLS connection between an Apache Kafka broker and his corresponding\nApache Zookeeper counter part.\n\nAs of today, this only covers using Zookeeper 3.5.5 with the upcoming Apache Kafka 2.4 version. Using it in earlier versions is not properly tested.\n\n## Run the playbook.\n\nTo run the playbook you need installed in your machine, docker, docker-compose.\n\nThe playbook can be started by running the _$> ./up_ script.\n\n\n### Configuration on Apache ZooKeeper\n\nRequired environment variables:\n\n```bash\nSERVER_JVMFLAGS=-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory\n````\n\nzoo.cfg file:\n\n```bash\nsecureClientPort=2182\nauthProvider.1=org.apache.zookeeper.server.auth.X509AuthenticationProvider\nssl.trustStore.location=/var/lib/secret/truststore.jks\nssl.trustStore.password=test1234\nssl.keyStore.location=/var/lib/secret/zookeeper.jks\nssl.keyStore.password=test1234\nssl.clientAuth=true\n```\n\n### Configuration for Apache Kafka\n\nRequired environment variables:\n\n```bash\nKAFKA_OPTS=-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location=/var/lib/secret/kafka.jks -Dzookeeper.ssl.keyStore.password=confluent -Dzookeeper.ssl.trustStore.location=/var/lib/secret/truststore.jks -Dzookeeper.ssl.trustStore.password=confluent\n```\n\nserver.properties file:\n\n```\nzookeeper.connect=zookeeper:2182\n```\n\nto use the secure port, a use can use both (but I would certainly not recommended as it water down security)\n\n## Things pending..\n\n* The current zookeeper migration tool works based on JAAS files, there is currently no option to set authentication in a different way. There is an issue open with Apache Kafka (https://issues.apache.org/jira/browse/KAFKA-8843) to fix this, as well as the required overall KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication, currently under discussion.\n\n* The https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication covers as well the challenge of configuring zookeeper TLS access, for the brokers, using environment variables. There is a change proposed to make things better.\n\n*NOTE*: This playbook utilised a custom made Apache Kafka docker image, build from a trunk snapshot the 22 of August 2019. Currently Apache Kafka 2.4 is still not released. Changing based images will be easy when an official confluent image is released. \n\n## Reference\n\n* https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide\n* https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication\n* https://issues.apache.org/jira/browse/KAFKA-8843\n* https://github.com/apache/kafka/commit/d67495d6a7f4c5f7e8736a25d6a11a1c1bef8d87\n"
},
{
"path": "apache-kafka-with-zk3.5-and-tls/docker-compose.yml",
"content": "version: '3'\nservices:\n zookeeper:\n build: zookeeper/\n container_name: zookeeper\n hostname: zookeeper\n restart: on-failure\n environment:\n - SERVER_JVMFLAGS=-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory\n volumes:\n - ./certs/zk-stores:/var/lib/secret\n\n kafka:\n build: kafka/\n container_name: kafka\n hostname: kafka\n depends_on:\n - zookeeper\n restart: on-failure\n volumes:\n - ./certs/kafka-stores:/var/lib/secret\n environment:\n - KAFKA_OPTS=-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location=/var/lib/secret/kafka.jks -Dzookeeper.ssl.keyStore.password=confluent -Dzookeeper.ssl.trustStore.location=/var/lib/secret/truststore.jks -Dzookeeper.ssl.trustStore.password=confluent\n ports:\n - 29092:29092\n"
},
{
"path": "apache-kafka-with-zk3.5-and-tls/kafka/Dockerfile",
"content": "FROM purbon/kafka\nMAINTAINER pere.urbon@gmail.com\nENV container docker\n\n# 1. Install openjdk\nRUN yum install -y java-11-openjdk\n\n# 2. Configure Kafka\nCOPY server.properties /etc/kafka/server.properties\n\nEXPOSE 9092\n\nCMD kafka-server-start.sh /etc/kafka/server.properties\n"
},
{
"path": "apache-kafka-with-zk3.5-and-tls/kafka/server.properties",
"content": " # Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# see kafka.server.KafkaConfig for additional details and defaults\n\n############################# Server Basics #############################\n\n# The id of the broker. This must be set to a unique integer for each broker.\nbroker.id=0\n\n############################# Socket Server Settings #############################\n\n# The address the socket server listens on. It will get the value returned from\n# java.net.InetAddress.getCanonicalHostName() if not configured.\n# FORMAT:\n# listeners = listener_name://host_name:port\n# EXAMPLE:\n# listeners = PLAINTEXT://your.host.name:9092\nlisteners=PLAINTEXT://kafka:9092,EXT_PLAINTEXT://localhost:29092\n\n# Hostname and port the broker will advertise to producers and consumers. If not set,\n# it uses the value for \"listeners\" if configured. Otherwise, it will use the value\n# returned from java.net.InetAddress.getCanonicalHostName().\nadvertised.listeners=PLAINTEXT://kafka:9092,EXT_PLAINTEXT://localhost:29092\n\n# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details\nlistener.security.protocol.map=PLAINTEXT:PLAINTEXT,EXT_PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL\n#security.inter.broker.protocol=SSL\n\n# The number of threads that the server uses for receiving requests from the network and sending responses to the network\nnum.network.threads=3\n\n# The number of threads that the server uses for processing requests, which may include disk I/O\nnum.io.threads=8\n\n# The send buffer (SO_SNDBUF) used by the socket server\nsocket.send.buffer.bytes=102400\n\n# The receive buffer (SO_RCVBUF) used by the socket server\nsocket.receive.buffer.bytes=102400\n\n# The maximum size of a request that the socket server will accept (protection against OOM)\nsocket.request.max.bytes=104857600\n\n\n############################# Log Basics #############################\n\n# A comma separated list of directories under which to store log files\nlog.dirs=/var/lib/kafka\n\n# The default number of log partitions per topic. More partitions allow greater\n# parallelism for consumption, but this will also result in more files across\n# the brokers.\nnum.partitions=1\n\n# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.\n# This value is recommended to be increased for installations with data dirs located in RAID array.\nnum.recovery.threads.per.data.dir=1\n\n############################# Internal Topic Settings #############################\n# The replication factor for the group metadata internal topics \"__consumer_offsets\" and \"__transaction_state\"\n# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.\noffsets.topic.replication.factor=1\ntransaction.state.log.replication.factor=1\ntransaction.state.log.min.isr=1\n\n############################# Log Flush Policy #############################\n\n# Messages are immediately written to the filesystem but by default we only fsync() to sync\n# the OS cache lazily. The following configurations control the flush of data to disk.\n# There are a few important trade-offs here:\n# 1. Durability: Unflushed data may be lost if you are not using replication.\n# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.\n# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.\n# The settings below allow one to configure the flush policy to flush data after a period of time or\n# every N messages (or both). This can be done globally and overridden on a per-topic basis.\n\n# The number of messages to accept before forcing a flush of data to disk\n#log.flush.interval.messages=10000\n\n# The maximum amount of time a message can sit in a log before we force a flush\n#log.flush.interval.ms=1000\n\n############################# Log Retention Policy #############################\n\n# The following configurations control the disposal of log segments. The policy can\n# be set to delete segments after a period of time, or after a given size has accumulated.\n# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens\n# from the end of the log.\n\n# The minimum age of a log file to be eligible for deletion due to age\nlog.retention.hours=168\n\n# A size-based retention policy for logs. Segments are pruned from the log unless the remaining\n# segments drop below log.retention.bytes. Functions independently of log.retention.hours.\n#log.retention.bytes=1073741824\n\n# The maximum size of a log segment file. When this size is reached a new log segment will be created.\nlog.segment.bytes=1073741824\n\n# The interval at which log segments are checked to see if they can be deleted according\n# to the retention policies\nlog.retention.check.interval.ms=300000\n\n############################# Zookeeper #############################\n\n# Zookeeper connection string (see zookeeper docs for details).\n# This is a comma separated host:port pairs, each corresponding to a zk\n# server. e.g. \"127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002\".\n# You can also append an optional chroot string to the urls to specify the\n# root directory for all kafka znodes.\nzookeeper.connect=zookeeper:2182\n\n# Timeout in ms for connecting to zookeeper\nzookeeper.connection.timeout.ms=6000\n\n##################### Confluent Metrics Reporter #######################\n# Confluent Control Center and Confluent Auto Data Balancer integration\n#\n# Uncomment the following lines to publish monitoring data for\n# Confluent Control Center and Confluent Auto Data Balancer\n# If you are using a dedicated metrics cluster, also adjust the settings\n# to point to your metrics kakfa cluster.\n#metric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter\n#confluent.metrics.reporter.bootstrap.servers=localhost:9092\n#\n# Uncomment the following line if the metrics cluster has a single broker\n#confluent.metrics.reporter.topic.replicas=1\n\n##################### Confluent Proactive Support ######################\n# If set to true, and confluent-support-metrics package is installed\n# then the feature to collect and report support metrics\n# (\"Metrics\") is enabled. If set to false, the feature is disabled.\n#\n#confluent.support.metrics.enable=false\n\n\n# The customer ID under which support metrics will be collected and\n# reported.\n#\n# When the customer ID is set to \"anonymous\" (the default), then only a\n# reduced set of metrics is being collected and reported.\n#\n# Confluent customers\n# -------------------\n# If you are a Confluent customer, then you should replace the default\n# value with your actual Confluent customer ID. Doing so will ensure\n# that additional support metrics will be collected and reported.\n#\n#confluent.support.customer.id=anonymous\n\n############################# Group Coordinator Settings #############################\n\n# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.\n# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.\n# The default value for this is 3 seconds.\n# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.\n# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.\ngroup.initial.rebalance.delay.ms=0\n\n\n# TLS Configuration\n#ssl.truststore.location=/var/lib/secret/truststore.jks\n#ssl.truststore.password=test1234\n#ssl.keystore.location=/var/lib/secret/server.keystore.jks\n#ssl.keystore.password=test1234\n#ssl.client.auth=required\n#authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer\n#super.users=User:CN=kafka.confluent.local,L=London,O=Confluent,C=UK;User:CN=schema-registry.confluent.local,L=London,O=Confluent,C=UK\n"
},
{
"path": "apache-kafka-with-zk3.5-and-tls/up",
"content": "#!/usr/bin/env bash\nset -e\n\nfunction gencert() {\n if [ -a $1.jks ];\n then\n echo \"The keystore $1.jks already exists\";\n exit;\n fi\n\n echo \"Creating keystore $1.jks with a certificate and a key-pair for CN $1\"\n keytool -keystore $1.jks -alias $1 -validity $VALIDITY -genkey -storepass $PASSWORD -keypass $PASSWORD -dname \"CN=$1,OU=kafka,O=confluent,L=MS,ST=Berlin,C=DE\"\n echo \"Creating a Certificate-Signing-Request for the generated certificate\"\n keytool -keystore $1.jks -alias $1 -certreq -file cert-file -storepass $PASSWORD\n echo \"Signing the Certificate-Signing-Request and adding an additional DNS-entry for localhost\"\n openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days $VALIDITY -CAcreateserial -passin pass:$PASSWORD -extensions SAN -extfile <(printf \"\\n[SAN]\\nsubjectAltName=DNS:$1,DNS:localhost\")\n echo \"Importing the root-certificate for the CA into the keystore $1.jks\"\n keytool -keystore $1.jks -alias CARoot -import -file ca-cert -storepass $PASSWORD -noprompt\n echo \"Importing the signed certificate for CN $1 into the keystore $1.jks\"\n keytool -keystore $1.jks -alias $1 -import -file cert-signed -storepass $PASSWORD\n echo \"Removing obsolet files...\"\n rm -v cert-file cert-signed\n}\n\nfunction gentruststore() {\n if [ -a ca-cert ];\n then\n echo \"The root-certificate for the CA already exists...\";\n else\n echo \"Creating a x509-certificate for the CA...\";\n openssl req -new -x509 -subj \"/C=DE/ST=Berlin/L=MS/O=confluent/OU=kafka/CN=Root-CA\" -keyout ca-key -out ca-cert -days $VALIDITY -passout pass:$PASSWORD\n fi\n #\n\n if [ -a truststore.jks ];\n then\n echo \"The keystore truststore.jks already exists!\";\n else\n echo \"Importing the root-certificate of the CA into truststore.jks...\"\n keytool -keystore truststore.jks -storepass $PASSWORD -alias CARoot -import -file ca-cert -noprompt\n fi\n}\n\nrm -rf certs\nrm -rf tmp-dir\nmkdir tmp-dir\nmkdir -p certs/kafka-stores\nmkdir -p certs/zk-stores\n\nVALIDITY=365\nPASSWORD=confluent\n\n\n(cd tmp-dir; gentruststore)\n\nhosts=( \"zookeeper\" \"client\" \"kafka\")\n\nfor host in \"${hosts[@]}\"\ndo\n (cd tmp-dir; gencert $host )\ndone\n\ncp tmp-dir/truststore.jks certs/kafka-stores\ncp tmp-dir/truststore.jks certs/zk-stores\ncp tmp-dir/zookeeper.jks certs/zk-stores\ncp tmp-dir/kafka.jks certs/kafka-stores\n\n# Starting docker-compose services\ndocker-compose up -d --build\n\necho \"Example configuration to access kafka:\"\necho \"-> docker-compose exec kafka kafka-topics.sh --bootstrap-server kafka:9092 --create --topic foo --partitions 1 --replication-factor 1\"\necho \"-> docker-compose exec kafka kafka-console-producer.sh --broker-list kafka:9092 --topic foo\"\necho \"-> docker-compose exec kafka kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic foo --from-beginning\"\n"
},
{
"path": "apache-kafka-with-zk3.5-and-tls/zookeeper/Dockerfile",
"content": "FROM purbon/zookeeper:3.5.5\nMAINTAINER pere.urbon@gmail.com\nENV container docker\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-11-openjdk\n\n\n# 3. Configure zookeeper\nCOPY zoo.cfg \"${ZK_HOME}/conf/zoo.cfg\"\n\n# 4. Add extra utility scripts\n\nENV PATH=\"/opt/tlsZkCli.sh:${PATH}\"\nCOPY tlsZkCli.sh /opt/tlsZkCli.sh\n\nEXPOSE 2182\n\nCMD zkServer.sh start-foreground\n"
},
{
"path": "apache-kafka-with-zk3.5-and-tls/zookeeper/tlsZkCli.sh",
"content": "##!/usr/bin/env bash\n\nexport CLIENT_JVMFLAGS=\"-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Dzookeeper.client.secure=true\n -Dzookeeper.ssl.keyStore.location=/var/lib/secret/zookeeper.jks\n -Dzookeeper.ssl.keyStore.password=confluent\n -Dzookeeper.ssl.trustStore.location=/var/lib/secret/truststore.jks\n -Dzookeeper.ssl.trustStore.password=confluent\"\n\nzkCli.sh -server $1\n"
},
{
"path": "apache-kafka-with-zk3.5-and-tls/zookeeper/zoo.cfg",
"content": "# The number of milliseconds of each tick\ntickTime=2000\n# The number of ticks that the initial\n# synchronization phase can take\ninitLimit=10\n# The number of ticks that can pass between\n# sending a request and getting an acknowledgement\nsyncLimit=5\n# the directory where the snapshot is stored.\n# do not use /tmp for storage, /tmp here is just\n# example sakes.\ndataDir=/tmp/zookeeper\n# the port at which the clients will connect\n#clientPort=2181\nsecureClientPort=2182\n# the maximum number of client connections.\n# increase this if you need to handle more clients\n#maxClientCnxns=60\n#\n# Be sure to read the maintenance section of the\n# administrator guide before turning on autopurge.\n#\n# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance\n#\n# The number of snapshots to retain in dataDir\n#autopurge.snapRetainCount=3\n# Purge task interval in hours\n# Set to \"0\" to disable auto purge feature\n#autopurge.purgeInterval=1\n\nauthProvider.1=org.apache.zookeeper.server.auth.X509AuthenticationProvider\nssl.trustStore.location=/var/lib/secret/truststore.jks\nssl.trustStore.password=confluent\nssl.keyStore.location=/var/lib/secret/zookeeper.jks\nssl.keyStore.password=confluent\n# This option is commented out only as an example of what is possible for the\n# SSL authentication. In a production environment this should be set as\n# here, with ssl.clientAuth=need\n#ssl.clientAuth=need\n"
},
{
"path": "auditlog/README.md",
"content": "# Kafka Audit Log\n\nThis playbook add an example of using the confluent audit log trail.\nThe present example works with SASL/SCRAM but this example can be extended to other authentication methods such as RBAC, other SASL flavours or TLS.\n\n## Playbook.\n\n1.- start all the components running the _./up_ script.\n\n```bash\n./up\nCreating zookeeper ... done\nCreating kafka ... done\nCompleted updating config for entity: user-principal 'kafka'.\nCompleted updating config for entity: user-principal 'consumer'.\nCompleted updating config for entity: user-principal 'producer'.\n[2020-05-12 12:20:50,405] WARN The configuration 'sasl.jaas.config' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)\nAdding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=*, patternType=LITERAL)`:\n \t(principal=User:producer, host=*, operation=DESCRIBE, permissionType=ALLOW)\n\t(principal=User:producer, host=*, operation=WRITE, permissionType=ALLOW)\n\t(principal=User:producer, host=*, operation=CREATE, permissionType=ALLOW)\n\n[2020-05-12 12:20:51,026] WARN The configuration 'sasl.jaas.config' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)\nCurrent ACLs for resource `ResourcePattern(resourceType=TOPIC, name=*, patternType=LITERAL)`:\n \t(principal=User:producer, host=*, operation=DESCRIBE, permissionType=ALLOW)\n\t(principal=User:producer, host=*, operation=CREATE, permissionType=ALLOW)\n\t(principal=User:producer, host=*, operation=WRITE, permissionType=ALLOW)\n\n[2020-05-12 12:20:53,986] WARN The configuration 'sasl.jaas.config' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)\nAdding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=*, patternType=LITERAL)`:\n \t(principal=User:consumer, host=*, operation=DESCRIBE, permissionType=ALLOW)\n\t(principal=User:consumer, host=*, operation=READ, permissionType=ALLOW)\n\nAdding ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:\n \t(principal=User:consumer, host=*, operation=READ, permissionType=ALLOW)\n\n[2020-05-12 12:20:54,538] WARN The configuration 'sasl.jaas.config' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)\nCurrent ACLs for resource `ResourcePattern(resourceType=TOPIC, name=*, patternType=LITERAL)`:\n \t(principal=User:producer, host=*, operation=CREATE, permissionType=ALLOW)\n\t(principal=User:producer, host=*, operation=DESCRIBE, permissionType=ALLOW)\n\t(principal=User:consumer, host=*, operation=DESCRIBE, permissionType=ALLOW)\n\t(principal=User:producer, host=*, operation=WRITE, permissionType=ALLOW)\n\t(principal=User:consumer, host=*, operation=READ, permissionType=ALLOW)\n\nCurrent ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:\n \t(principal=User:consumer, host=*, operation=READ, permissionType=ALLOW)\n\n[2020-05-12 12:20:57,354] WARN The configuration 'sasl.jaas.config' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)\nAdding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=confluent-audit-log-events, patternType=PREFIXED)`:\n \t(principal=User:confluent-audit, host=*, operation=DESCRIBE, permissionType=ALLOW)\n\t(principal=User:confluent-audit, host=*, operation=WRITE, permissionType=ALLOW)\n\t(principal=User:confluent-audit, host=*, operation=CREATE, permissionType=ALLOW)\n\n[2020-05-12 12:20:57,928] WARN The configuration 'sasl.jaas.config' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)\nCurrent ACLs for resource `ResourcePattern(resourceType=TOPIC, name=confluent-audit-log-events, patternType=PREFIXED)`:\n \t(principal=User:confluent-audit, host=*, operation=WRITE, permissionType=ALLOW)\n\t(principal=User:confluent-audit, host=*, operation=DESCRIBE, permissionType=ALLOW)\n\t(principal=User:confluent-audit, host=*, operation=CREATE, permissionType=ALLOW)\n\nExample configuration:\n-> docker-compose exec kafka kafka-console-producer --broker-list kafka:9092 --producer.config /etc/kafka/producer-user.properties --topic test\n-> docker-compose exec kafka kafka-console-consumer --bootstrap-server kafka:9092 --consumer.config /etc/kafka/consumer-user.properties --topic test --from-beginning\n```\n\n2.- Explore the currently created topics.\n\n```\n./scripts/describe-topics.sh\n[2020-05-12 12:21:55,868] WARN The configuration 'sasl.jaas.config' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig)\nTopic: _confluent-license\tPartitionCount: 1\tReplicationFactor: 1\tConfigs: min.insync.replicas=1,cleanup.policy=compact\n\tTopic: _confluent-license\tPartition: 0\tLeader: 1\tReplicas: 1\tIsr: 1\tOffline:\nTopic: __confluent.support.metrics\tPartitionCount: 1\tReplicationFactor: 1\tConfigs: retention.ms=31536000000\n\tTopic: __confluent.support.metrics\tPartition: 0\tLeader: 1\tReplicas: 1\tIsr: 1\tOffline:\nTopic: confluent-audit-log-events\tPartitionCount: 12\tReplicationFactor: 1\tConfigs: retention.ms=7776000000,message.timestamp.type=CreateTime,retention.bytes=-1,segment.ms=14400000\n\tTopic: confluent-audit-log-events\tPartition: 0\tLeader: 1\tReplicas: 1\tIsr: 1\tOffline:\n\tTopic: confluent-audit-log-events\tPartition: 1\tLeader: 1\tReplicas: 1\tIsr: 1\tOffline:\n\tTopic: confluent-audit-log-events\tPartition: 2\tLeader: 1\tReplicas: 1\tIsr: 1\tOffline:\n\tTopic: confluent-audit-log-events\tPartition: 3\tLeader: 1\tReplicas: 1\tIsr: 1\tOffline:\n\tTopic: confluent-audit-log-events\tPartition: 4\tLeader: 1\tReplicas: 1\tIsr: 1\tOffline:\n\tTopic: confluent-audit-log-events\tPartition: 5\tLeader: 1\tReplicas: 1\tIsr: 1\tOffline:\n\tTopic: confluent-audit-log-events\tPartition: 6\tLeader: 1\tReplicas: 1\tIsr: 1\tOffline:\n\tTopic: confluent-audit-log-events\tPartition: 7\tLeader: 1\tReplicas: 1\tIsr: 1\tOffline:\n\tTopic: confluent-audit-log-events\tPartition: 8\tLeader: 1\tReplicas: 1\tIsr: 1\tOffline:\n\tTopic: confluent-audit-log-events\tPartition: 9\tLeader: 1\tReplicas: 1\tIsr: 1\tOffline:\n\tTopic: confluent-audit-log-events\tPartition: 10\tLeader: 1\tReplicas: 1\tIsr: 1\tOffline:\n\tTopic: confluent-audit-log-events\tPartition: 11\tLeader: 1\tReplicas: 1\tIsr: 1\tOffline:\n```\n\n3.- Explore the audit log topics\n\n```\n./scripts/explore-audit-topic.sh\n\n\n```\nempty at the beginning.\n\nKeep this open and it will start showing the generated events as we're issuing them.\n\n\n4.- Create some topics and acls.\n\n```\n./scripts/create-topics.sh\nCreate topic foo with User:kafka\nNOTE: this topic creation will be ignored because uses a user inside the ignore list.\nCreated topic foo.\nCreate topic bar with User:producer\nNOTE: This action will be noted in the audit log.\n\nCreated topic bar.\nAdding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=bar, patternType=LITERAL)`:\n \t(principal=User:producer, host=*, operation=ALTER_CONFIGS, permissionType=ALLOW)\n\nCurrent ACLs for resource `ResourcePattern(resourceType=TOPIC, name=bar, patternType=LITERAL)`:\n \t(principal=User:producer, host=*, operation=ALTER_CONFIGS, permissionType=ALLOW)\nAdding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=bar, patternType=LITERAL)`:\n \t(principal=User:producer, host=*, operation=DELETE, permissionType=ALLOW)\nCurrent ACLs for resource `ResourcePattern(resourceType=TOPIC, name=bar, patternType=LITERAL)`:\n \t(principal=User:producer, host=*, operation=ALTER_CONFIGS, permissionType=ALLOW)\n\t(principal=User:producer, host=*, operation=DELETE, permissionType=ALLOW)\n\nChange of a configuration\nNOTE: This action will be noted in the audit log.\n\nCompleted updating config for topic bar.\n```\n\nNow the audit log topic should reflect the information about the generated actions.\n\n```\n./scripts/explore-audit-topic.sh\n{\"data\":{\"serviceName\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA\",\"methodName\":\"kafka.CreateTopics\",\"resourceName\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA/topic=bar\",\"authenticationInfo\":{\"principal\":\"User:producer\"},\"authorizationInfo\":{\"granted\":true,\"operation\":\"Create\",\"resourceType\":\"Topic\",\"resourceName\":\"bar\",\"patternType\":\"LITERAL\",\"aclAuthorization\":{\"permissionType\":\"ALLOW\",\"host\":\"*\"}},\"request\":{\"correlation_id\":\"4\",\"client_id\":\"adminclient-1\"},\"requestMetadata\":{\"client_address\":\"/172.27.0.3\"}},\"id\":\"b17cc9c7-96f0-413a-b94c-124e21834a55\",\"source\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA\",\"specversion\":\"0.3\",\"type\":\"io.confluent.kafka.server/authorization\",\"time\":\"2020-05-12T12:24:37.838Z\",\"datacontenttype\":\"application/json\",\"subject\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA/topic=bar\",\"confluentRouting\":{\"route\":\"confluent-audit-log-events\"}}\n{\"data\":{\"serviceName\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA\",\"methodName\":\"kafka.IncrementalAlterConfigs\",\"resourceName\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA/topic=bar\",\"authenticationInfo\":{\"principal\":\"User:producer\"},\"authorizationInfo\":{\"granted\":true,\"operation\":\"AlterConfigs\",\"resourceType\":\"Topic\",\"resourceName\":\"bar\",\"patternType\":\"LITERAL\",\"aclAuthorization\":{\"permissionType\":\"ALLOW\",\"host\":\"*\"}},\"request\":{\"correlation_id\":\"4\",\"client_id\":\"adminclient-1\"},\"requestMetadata\":{\"client_address\":\"/172.27.0.3\"}},\"id\":\"93e94659-8b45-4f44-b691-f284192ebe42\",\"source\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA\",\"specversion\":\"0.3\",\"type\":\"io.confluent.kafka.server/authorization\",\"time\":\"2020-05-12T12:24:47.700Z\",\"datacontenttype\":\"application/json\",\"subject\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA/topic=bar\",\"confluentRouting\":{\"route\":\"confluent-audit-log-events\"}}\n```\n\n5.- Write some messages\n\n```\n./scripts/write-msg.sh bar\nWrite messages to topic bar\n```\n\nMore messages coming into the audit log.\n\n```\n{\"data\":{\"serviceName\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA\",\"methodName\":\"kafka.Produce\",\"resourceName\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA/topic=bar\",\"authenticationInfo\":{\"principal\":\"User:producer\"},\"authorizationInfo\":{\"granted\":true,\"operation\":\"Write\",\"resourceType\":\"Topic\",\"resourceName\":\"bar\",\"patternType\":\"LITERAL\",\"aclAuthorization\":{\"permissionType\":\"ALLOW\",\"host\":\"*\"}},\"request\":{\"correlation_id\":\"6\",\"client_id\":\"rdkafka\"},\"requestMetadata\":{\"client_address\":\"/172.27.0.4\"}},\"id\":\"7789d492-df1c-404a-b494-3dc44fb01b24\",\"source\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA\",\"specversion\":\"0.3\",\"type\":\"io.confluent.kafka.server/authorization\",\"time\":\"2020-05-12T12:26:49.353Z\",\"datacontenttype\":\"application/json\",\"subject\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA/topic=bar\",\"confluentRouting\":{\"route\":\"confluent-audit-log-events\"}}\n```\n\n6.- Delete of messages\n\n```\n./scripts/delete-records.sh\nExecuting records delete operation\nRecords delete operation completed:\npartition: bar-0\tlow_watermark: 3\n```\n\nnew messages in the audit trail.\n\n```\n{\"data\":{\"serviceName\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA\",\"methodName\":\"kafka.DeleteRecords\",\"resourceName\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA/topic=bar\",\"authenticationInfo\":{\"principal\":\"User:producer\"},\"authorizationInfo\":{\"granted\":true,\"operation\":\"Delete\",\"resourceType\":\"Topic\",\"resourceName\":\"bar\",\"patternType\":\"LITERAL\",\"aclAuthorization\":{\"permissionType\":\"ALLOW\",\"host\":\"*\"}},\"request\":{\"correlation_id\":\"4\",\"client_id\":\"adminclient-1\"},\"requestMetadata\":{\"client_address\":\"/172.27.0.3\"}},\"id\":\"f6664ede-fbd4-4425-873a-d31df5eb0b7f\",\"source\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA\",\"specversion\":\"0.3\",\"type\":\"io.confluent.kafka.server/authorization\",\"time\":\"2020-05-12T12:27:34.425Z\",\"datacontenttype\":\"application/json\",\"subject\":\"crn:///kafka=STOiZ_jWTxqgum3T5zEoqA/topic=bar\",\"confluentRouting\":{\"route\":\"confluent-audit-log-events\"}}\n```\n\n## More information\n\nThis is only a summary and playbook of this functionality, more intel can be found in the reference documentation.\n\n1.- https://docs.confluent.io/current/security/audit-logs.html\n"
},
{
"path": "auditlog/config/delete-records.json",
"content": "{\n \"partitions\": [\n {\n \"topic\": \"bar\",\n \"partition\": 0,\n \"offset\": 3\n }\n ],\n \"version\": 1\n}\n"
},
{
"path": "auditlog/data/my_msgs.txt",
"content": "This is a message\nThis is another message\nAbracadabra\n"
},
{
"path": "auditlog/docker-compose.yml",
"content": "version: \"2\"\nservices:\n zookeeper:\n image: confluentinc/cp-zookeeper:5.5.0\n hostname: zookeeper\n container_name: zookeeper\n volumes:\n - ./zookeeper:/etc/kafka\n ports:\n - 2181:2181\n environment:\n ZOOKEEPER_CLIENT_PORT: 2181\n KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/zookeeper.sasl.jaas.config\n -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider\n\n\n kafka:\n image: confluentinc/cp-server:5.5.0\n container_name: kafka\n hostname: kafka\n depends_on:\n - zookeeper\n volumes:\n - ./kafka:/etc/kafka\n - ./config:/tmp/config\n ports:\n - 9092:9092\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: \"zookeeper:2181\"\n KAFKA_ADVERTISED_LISTENERS: \"SASL_PLAINTEXT://kafka:9092\"\n KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS: \"1\"\n KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: \"1\"\n KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: \"1\"\n KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: \"1\"\n KAFKA_SASL_ENABLED_MECHANISMS: \"SCRAM-SHA-256\"\n KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: \"SCRAM-SHA-256\"\n KAFKA_SECURITY_INTER_BROKER_PROTOCOL: \"SASL_PLAINTEXT\"\n KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: \"false\"\n KAFKA_OPTS: \"-Djava.security.auth.login.config=/etc/kafka/kafka.sasl.jaas.config\"\n KAFKA_SUPER_USERS: \"User:kafka\"\n KAFKA_ZOOKEEPER_SET_ACL: \"true\"\n KAFKA_AUTHORIZER_CLASS_NAME: io.confluent.kafka.security.authorizer.ConfluentServerAuthorizer\n KAFKA_CONFLUENT_SECURITY_EVENT_ROUTER_CONFIG: \"{\\\"routes\\\":{\\\"crn:///kafka=*/group=*\\\":{\\\"consume\\\":{\\\"allowed\\\":\\\"confluent-audit-log-events\\\",\\\"denied\\\":\\\"confluent-audit-log-events\\\"}},\\\"crn:///kafka=*/topic=*\\\":{\\\"produce\\\":{\\\"allowed\\\":\\\"confluent-audit-log-events\\\",\\\"denied\\\":\\\"confluent-audit-log-events\\\"},\\\"consume\\\":{\\\"allowed\\\":\\\"confluent-audit-log-events\\\",\\\"denied\\\":\\\"confluent-audit-log-events\\\"}}},\\\"destinations\\\":{\\\"topics\\\":{\\\"confluent-audit-log-events\\\":{\\\"retention_ms\\\":7776000000}}},\\\"default_topics\\\":{\\\"allowed\\\":\\\"confluent-audit-log-events\\\",\\\"denied\\\":\\\"confluent-audit-log-events\\\"},\\\"excluded_principals\\\":[\\\"User:kafka\\\",\\\"User:ANONYMOUS\\\"]}\"\n"
},
{
"path": "auditlog/example-config.json",
"content": "{\n\t\"routes\": {\n\t\t\"crn:///kafka=*/group=*\": {\n\t\t\t\"consume\": {\n\t\t\t\t\"allowed\": \"confluent-audit-log-events\",\n\t\t\t\t\"denied\": \"confluent-audit-log-events\"\n\t\t\t}\n\t\t},\n\t\t\"crn:///kafka=*/topic=*\": {\n\t\t\t\"produce\": {\n\t\t\t\t\"allowed\": \"confluent-audit-log-events\",\n\t\t\t\t\"denied\": \"confluent-audit-log-events\"\n\t\t\t},\n\t\t\t\"consume\": {\n\t\t\t\t\"allowed\": \"confluent-audit-log-events\",\n\t\t\t\t\"denied\": \"confluent-audit-log-events\"\n\t\t\t}\n\t\t}\n\t},\n\t\"destinations\": {\n\t\t\"topics\": {\n\t\t\t\"confluent-audit-log-events\": {\n\t\t\t\t\"retention_ms\": 7776000000\n\t\t\t}\n\t\t}\n\t},\n\t\"default_topics\": {\n\t\t\"allowed\": \"confluent-audit-log-events\",\n\t\t\"denied\": \"confluent-audit-log-events\"\n\t},\n\t\"excluded_principals\": [\"User:kafka\", \"User:ANONYMOUS\"]\n}\n"
},
{
"path": "auditlog/kafka/consumer-user.properties",
"content": "sasl.mechanism=SCRAM-SHA-256\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"consumer\" \\\n password=\"consumerpass\";\n\n"
},
{
"path": "auditlog/kafka/kafka-user.properties",
"content": "sasl.mechanism=SCRAM-SHA-256\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"kafka\" \\\n password=\"kafka\";\n\n"
},
{
"path": "auditlog/kafka/kafka.properties",
"content": "broker.id=1\nadvertised.listeners=SASL_PLAINTEXT://kafka:9092\noffsets.topic.replication.factor=1\nallow.everyone.if.no.acl.found=false\nzookeeper.connect=zookeeper:2181\nsecurity.inter.broker.protocol=SASL_PLAINTEXT\nauthorizer.class.name=io.confluent.kafka.security.authorizer.ConfluentServerAuthorizer\nlog.dirs=/var/lib/kafka/data\nconfluent.security.event.router.config={\"routes\":{\"crn:///kafka=*/group=*\":{\"consume\":{\"allowed\":\"confluent-audit-log-events\",\"denied\":\"confluent-audit-log-events\"}},\"crn:///kafka=*/topic=*\":{\"produce\":{\"allowed\":\"confluent-audit-log-events\",\"denied\":\"confluent-audit-log-events\"},\"consume\":{\"allowed\":\"confluent-audit-log-events\",\"denied\":\"confluent-audit-log-events\"}}},\"destinations\":{\"topics\":{\"confluent-audit-log-events\":{\"retention_ms\":7776000000}}},\"default_topics\":{\"allowed\":\"confluent-audit-log-events\",\"denied\":\"confluent-audit-log-events\"},\"excluded_principals\":[\"User:kafka\",\"User:ANONYMOUS\"]}\nlisteners=SASL_PLAINTEXT://0.0.0.0:9092\nzookeeper.set.acl=true\nsuper.users=User:kafka\noffsets.topic.num.partitions=1\nsasl.enabled.mechanisms=SCRAM-SHA-256\ntransaction.state.log.replication.factor=1\nsasl.mechanism.inter.broker.protocol=SCRAM-SHA-256\nconfluent.license.topic.replication.factor=1\n-%}\n\n"
},
{
"path": "auditlog/kafka/kafka.sasl.jaas.config",
"content": "KafkaServer {\n org.apache.kafka.common.security.scram.ScramLoginModule required\n username=\"kafka\"\n password=\"kafka\";\n};\nClient {\n org.apache.zookeeper.server.auth.DigestLoginModule required\n username=\"admin\"\n password=\"password\";\n};\n"
},
{
"path": "auditlog/kafka/log4j.properties",
"content": "\nlog4j.rootLogger=INFO, stdout\n\nlog4j.appender.stdout=org.apache.log4j.ConsoleAppender\nlog4j.appender.stdout.layout=org.apache.log4j.PatternLayout\nlog4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n\n\n\nlog4j.logger.kafka.authorizer.logger=WARN\nlog4j.logger.kafka.log.LogCleaner=INFO\nlog4j.logger.kafka.producer.async.DefaultEventHandler=DEBUG\nlog4j.logger.kafka.controller=TRACE\nlog4j.logger.kafka.network.RequestChannel$=WARN\nlog4j.logger.kafka.request.logger=WARN\nlog4j.logger.state.change.logger=TRACE\nlog4j.logger.kafka=INFO\n"
},
{
"path": "auditlog/kafka/producer-user.properties",
"content": "sasl.mechanism=SCRAM-SHA-256\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"producer\" \\\n password=\"producerpass\";\n\n"
},
{
"path": "auditlog/kafka/tools-log4j.properties",
"content": "\nlog4j.rootLogger=WARN, stderr\n\nlog4j.appender.stderr=org.apache.log4j.ConsoleAppender\nlog4j.appender.stderr.layout=org.apache.log4j.PatternLayout\nlog4j.appender.stderr.layout.ConversionPattern=[%d] %p %m (%c)%n\nlog4j.appender.stderr.Target=System.err"
},
{
"path": "auditlog/scripts/create-topics.sh",
"content": "#!/usr/bin/env bash\n\necho \"Create topic foo with User:kafka\"\necho \"NOTE: this topic creation will be ignored because uses a user inside the ignore list.\"\necho\ndocker exec kafka kafka-topics --bootstrap-server kafka:9092 \\\n --command-config /etc/kafka/kafka-user.properties \\\n --create --topic foo --replication-factor 1 --partitions 1\nsleep 1\necho \"Create topic bar with User:producer\"\necho \"NOTE: This action will be noted in the audit log.\"\necho\ndocker exec kafka kafka-topics --bootstrap-server kafka:9092 \\\n --command-config /etc/kafka/producer-user.properties \\\n --create --topic bar --replication-factor 1 --partitions 1\n\n## Add extra ACLs need to handle the topic bar\ndocker exec kafka kafka-acls --bootstrap-server kafka:9092 \\\n --command-config /etc/kafka/kafka-user.properties \\\n --add --allow-principal User:producer --operation AlterConfigs \\\n --topic \"bar\"\n\ndocker exec kafka kafka-acls --bootstrap-server kafka:9092 \\\n --command-config /etc/kafka/kafka-user.properties \\\n --add --allow-principal User:producer --operation Delete \\\n --topic \"bar\"\n\nsleep 1\n\necho \"Change of a configuration\"\necho \"NOTE: This action will be noted in the audit log.\"\necho\ndocker exec kafka kafka-configs --bootstrap-server kafka:9092 \\\n --topic bar --add-config retention.ms=2592000001 \\\n --alter --command-config /etc/kafka/producer-user.properties\n"
},
{
"path": "auditlog/scripts/delete-records.sh",
"content": "#!/usr/bin/env bash\n\ndocker exec kafka kafka-delete-records --bootstrap-server kafka:9092 \\\n --command-config /etc/kafka/producer-user.properties \\\n --offset-json-file /tmp/config/delete-records.json\n"
},
{
"path": "auditlog/scripts/describe-topics.sh",
"content": "#!/usr/bin/env bash\n\ndocker exec kafka kafka-topics --bootstrap-server kafka:9092 --command-config /etc/kafka/kafka-user.properties --describe\n"
},
{
"path": "auditlog/scripts/explore-audit-topic.sh",
"content": "#!/usr/bin/env bash\n\nTOPIC=\"confluent-audit-log-events\"\ndocker-compose exec kafka kafka-console-consumer --bootstrap-server kafka:9092 \\\n --consumer.config /etc/kafka/kafka-user.properties \\\n --topic $TOPIC --from-beginning\n"
},
{
"path": "auditlog/scripts/write-msg.sh",
"content": "#!/usr/bin/env bash\n\nPWD=`pwd`\ntopic=$1\nnetwork=\"auditlog_default\"\n\nUSERNAME=producer\nPASSWORD=producerpass\n\necho \"Write messages to topic $1\"\n\ndocker run --network $network \\\n --volume $PWD/data/my_msgs.txt:/data/my_msgs.txt \\\n confluentinc/cp-kafkacat \\\n kafkacat -b kafka:9092 \\\n -t $topic \\\n -X security.protocol=SASL_PLAINTEXT -X sasl.mechanisms=SCRAM-SHA-256 -X sasl.username=$USERNAME -X sasl.password=$PASSWORD \\\n -P -l /data/my_msgs.txt\n"
},
{
"path": "auditlog/up",
"content": "#!/usr/bin/env bash\n\ndocker-compose up -d\n\ndocker-compose exec kafka kafka-configs --zookeeper zookeeper:2181 --alter --add-config 'SCRAM-SHA-256=[password=kafka],SCRAM-SHA-512=[password=kafka]' --entity-type users --entity-name kafka\ndocker-compose exec zookeeper kafka-configs --zookeeper zookeeper:2181 --alter --add-config 'SCRAM-SHA-256=[password=consumerpass],SCRAM-SHA-512=[password=consumerpass]' --entity-type users --entity-name consumer\ndocker-compose exec zookeeper kafka-configs --zookeeper zookeeper:2181 --alter --add-config 'SCRAM-SHA-256=[password=producerpass],SCRAM-SHA-512=[password=producerpass]' --entity-type users --entity-name producer\n\n# ACLs\ndocker-compose exec kafka kafka-acls --bootstrap-server kafka:9092 --command-config /etc/kafka/kafka-user.properties --add --allow-principal User:producer --producer --topic=*\ndocker-compose exec kafka kafka-acls --bootstrap-server kafka:9092 --command-config /etc/kafka/kafka-user.properties --add --allow-principal User:consumer --consumer --topic=* --group=*\ndocker-compose exec kafka kafka-acls --bootstrap-server kafka:9092 --command-config /etc/kafka/kafka-user.properties --add --allow-principal User:confluent-audit --producer --topic confluent-audit-log-events --resource-pattern-type prefixed\n\necho \"Example configuration:\"\necho \"-> docker-compose exec kafka kafka-console-producer --broker-list kafka:9092 --producer.config /etc/kafka/producer-user.properties --topic test\"\necho \"-> docker-compose exec kafka kafka-console-consumer --bootstrap-server kafka:9092 --consumer.config /etc/kafka/consumer-user.properties --topic test --from-beginning\"\n"
},
{
"path": "auditlog/zookeeper/log4j.properties",
"content": "\nlog4j.rootLogger=INFO, stdout\n\nlog4j.appender.stdout=org.apache.log4j.ConsoleAppender\nlog4j.appender.stdout.layout=org.apache.log4j.PatternLayout\nlog4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n\n\n"
},
{
"path": "auditlog/zookeeper/tools-log4j.properties",
"content": "\nlog4j.rootLogger=WARN, stderr\n\nlog4j.appender.stderr=org.apache.log4j.ConsoleAppender\nlog4j.appender.stderr.layout=org.apache.log4j.PatternLayout\nlog4j.appender.stderr.layout.ConversionPattern=[%d] %p %m (%c)%n\nlog4j.appender.stderr.Target=System.err"
},
{
"path": "auditlog/zookeeper/zookeeper.properties",
"content": "\ndataDir=/var/lib/zookeeper/data\ndataLogDir=/var/lib/zookeeper/log\n\nclientPort=2181\n\n\n"
},
{
"path": "auditlog/zookeeper/zookeeper.sasl.jaas.config",
"content": "Server {\n org.apache.zookeeper.server.auth.DigestLoginModule required\n user_admin=\"password\";\n};\nClient {\n org.apache.zookeeper.server.auth.DigestLoginModule required\n username=\"admin\"\n password=\"password\";\n};\n"
},
{
"path": "ca-builder-scripts/.gitignore",
"content": "tmp-certs/\nstores\nlegacy/\n\n## remove from git the generated CA files\n\nca/\n"
},
{
"path": "ca-builder-scripts/README.md",
"content": "# Building a CA with OpenSSL\n\nThis is a collection is scripts useful to generated a local CA setup. While the PKI could be set in different ways, for this example\nwe generate:\n\n* A root CA identity, for example for your mother company.\n* An intermediate CA identity, for example the one generated for your department or smaller company.\n* And then the client certificates.\n\nAll the scripts are wrapping openssl to generate the required structures.\n\n\n## Commands\n\nA collection of scripts are provided to generate any of the required steps to build this CA.\n\n*IMPORTANT:* This scripts set a default password for the CA certs, this password is: __confluent__ . You should change it.\n\n### Building the root CA\n\nTo construct the root CA setup, you can run the script:\n\n```bash\n./utils/build-ca.sh\n```\n\nthis script will generate the default directory structure for the CA, including the root certificate for your authority. After the execution you should see a directory structure like this:\n\n```bash\n➜ ca-builder-scripts git:(ca-builder-scripts) ✗ ls -la ca\ntotal 48\ndrwxr-xr-x 13 pere staff 416 3 May 16:33 .\ndrwxr-xr-x 16 pere staff 512 3 May 16:32 ..\ndrwxr-xr-x 3 pere staff 96 3 May 16:33 certs\ndrwxr-xr-x 2 pere staff 64 3 May 16:32 crl\n-rw-r--r-- 1 pere staff 97 3 May 16:33 index.txt\n-rw-r--r-- 1 pere staff 21 3 May 16:33 index.txt.attr\ndrwxr-xr-x 3 pere staff 96 3 May 16:33 newcerts\n-rw-r--r-- 1 pere staff 4117 3 May 16:32 openssl.cnf\ndrwx------ 3 pere staff 96 3 May 16:32 private\n-rw-r--r-- 1 pere staff 5 3 May 16:33 serial\n```\n\n*NOTE*: This script sets a default password for the root certificate, change it if you require to have another one.\n\n### Building the intermediate CA.\n\nOnce the main CA structure is created, you need to create the intermediate CA, for this you can use this script:\n\n```bash\n./utils/build-intermediate-ca.sh\n```\n\nOnce the script is run, you should see a directory structure like this:\n\n```bash\n➜ ca-builder-scripts git:(ca-builder-scripts) ✗ ls -la ca/intermediate\ntotal 80\ndrwxr-xr-x 16 pere staff 512 3 May 17:22 .\ndrwxr-xr-x 13 pere staff 416 3 May 16:33 ..\ndrwxr-xr-x 5 pere staff 160 3 May 16:34 certs\ndrwxr-xr-x 3 pere staff 96 3 May 17:21 crl\n-rw-r--r-- 1 pere staff 5 3 May 17:22 crlnumber\ndrwxr-xr-x 4 pere staff 128 3 May 16:34 csr\n-rw-r--r-- 1 pere staff 109 3 May 17:20 index.txt\n-rw-r--r-- 1 pere staff 21 3 May 17:20 index.txt.attr\ndrwxr-xr-x 3 pere staff 96 3 May 16:34 newcerts\n-rw-r--r-- 1 pere staff 4328 3 May 16:33 openssl.cnf\ndrwx------ 4 pere staff 128 3 May 16:33 private\n-rw-r--r-- 1 pere staff 5 3 May 16:34 serial\n```\n*NOTE*: This script sets a default password for the certificate, change it if you require to have another one.\n\n### Generating an end user certificate\n\nOnce the full CA is setup, next step is to generate end user certificates, to do this you can use a command that look like:\n\n```bash\n./create-pair-certs.sh kafka.confluent.local server_cert\n```\nwhere the first parameter is the certificate name and the second is the extension being used. For this CA we support server_certs and usr_cert. See the [configs/](configs/) directory for details of the configuration.\n\n### revoke certs\n\nA common process in any CA is to revoke certificates, in with this scripts you can do it like this:\n\n```bash\n./revoke-cert.sh kafka.confluent.local\n```\n\nthis command will revoke a certificate with the name _kakfa.confluent.local_.\n\nOnce this command is run, you should an update in the intermediate CA text db like this:\n\n```bash\n➜ ca-builder-scripts git:(ca-builder-scripts) ✗ cat ca/intermediate/index.txt\nR\t200512143408Z\t190503152037Z\t1000\tunknown\t/C=DE/ST=Berlin/L=Berlin/O=Confluent Ltd/CN=kafka.confluent.local\n```\n\nthis means this cert is revoked, so no longer valid\n\n\n## create certificate revocation lists\n\nTo revoke a cert is nice, but you need to announce this to the world, for this you need to create a list of revoked certificates. This you can do using this script:\n\n```bash\n./create-crl.sh\n```\n\nOnce this is run, there will be a new file being created under\n\n```bash\n➜ ca-builder-scripts git:(ca-builder-scripts) ✗ ls ca/intermediate/crl\nintermediate.crl.pem\n```\n\nthat will contain the list of revoked certs, this can be used then as part of your distribution points list, to inform clients of the CA which identities are being revoked.\n\n\n## Common errors\n\n> error 20 at 0 depth lookup:unable to get local issuer certificate\n\ncould not find the original file, paths to cerfiticates CA is wrong.\n\n> TXT_DB error number 2 failed to update database\n\nBecause you have generated your own self signed certificate with the same CN (Common Name) information that the CA certificate that you’ve generated before.\n\nEnter another Common Name.\n"
},
{
"path": "ca-builder-scripts/build-a-batch-of-certs.sh",
"content": "#!/usr/bin/env bash\n\ninput=$1\nwhile IFS= read -r line\ndo\n fields=($(echo $line | tr \",\" \"\\n\"))\n #./support-scripts/create-cert.sh ${fields[0]} ${fields[1]}\n echo \"./support-scripts/create-cert.sh ${fields[0]} ${fields[1]}\"\ndone < \"$input\"\n"
},
{
"path": "ca-builder-scripts/build-a-batch-of-stores.sh",
"content": "#!/usr/bin/env bash\n\nDEFAULT_PASSWORD=${2:-confluent}\n\nif [ -z \"${CA_ROOT_DIR+x}\" ];\nthen\nCA_ROOT_DIR='.'\nfi\n\nCA_CERT=\"$CA_ROOT_DIR/ca/certs/ca.cert.pem\"\nCA_KEY=\"$CA_ROOT_DIR/ca/private/ca.key.pem\"\n\nINT_CA_CERT=\"$CA_ROOT_DIR/ca/intermediate/certs/intermediate.cert.pem\"\nINT_CA_KEY=\"$CA_ROOT_DIR/ca/intermediate/private/intermediate.key.pem\"\n\nfunction build_store {\n cert_name=$1\n store_type=$2\n\n CERT_PATH=\"$CA_ROOT_DIR/ca/intermediate/certs/$cert_name.cert.pem\"\n KEY_PATH=\"$CA_ROOT_DIR/ca/intermediate/private/$cert_name.key.pem\"\n openssl pkcs12 -export -in $CERT_PATH -inkey $KEY_PATH -passin pass:$DEFAULT_PASSWORD -passout pass:$DEFAULT_PASSWORD -name $cert_name -out tmp-certs/$cert_name.p12\n sleep 1\n ## build keystore and truststores\n keytool -noprompt -importkeystore -deststorepass $DEFAULT_PASSWORD -destkeystore stores/$store_type.keystore -srckeystore tmp-certs/$cert_name.p12 -srcstorepass $DEFAULT_PASSWORD -storepass $DEFAULT_PASSWORD -keypass $DEFAULT_PASSWORD -srcstoretype PKCS12 -deststoretype pkcs12\n\n openssl pkcs12 -export -in $CA_CERT -inkey $CA_KEY -passin pass:$DEFAULT_PASSWORD -passout pass:$DEFAULT_PASSWORD -name 'ca' -out tmp-certs/ca.p12\n sleep 1\n openssl pkcs12 -export -in $INT_CA_CERT -inkey $INT_CA_KEY -passin pass:$DEFAULT_PASSWORD -passout pass:$DEFAULT_PASSWORD -name 'intermediate-ca' -out tmp-certs/inter-ca.p12\n sleep 1\n\n keytool -noprompt -importkeystore -deststorepass $DEFAULT_PASSWORD -destkeystore stores/$store_type.truststore -srckeystore tmp-certs/ca.p12 -srcstorepass $DEFAULT_PASSWORD -srcstoretype PKCS12 -storepass $DEFAULT_PASSWORD -keypass $DEFAULT_PASSWORD -deststoretype pkcs12\n sleep 1\n keytool -noprompt -importkeystore -deststorepass $DEFAULT_PASSWORD -destkeystore stores/$store_type.truststore -srckeystore tmp-certs/inter-ca.p12 -srcstorepass $DEFAULT_PASSWORD -srcstoretype PKCS12 -storepass $DEFAULT_PASSWORD -keypass $DEFAULT_PASSWORD -deststoretype pkcs12\n}\n\n## buildind stores for the brokers\n\nmkdir -p stores tmp-certs\n\nCONFIG_FILE=$1\nwhile read data; do\n fields=($(echo $data | tr \",\" \"\\n\"))\n echo \"Building a store for ${fields[0]} and ${fields[1]}\"\n build_store \"${fields[0]}\" \"${fields[1]}\"\ndone <$CONFIG_FILE\n\nrm -rf temp-certs\n"
},
{
"path": "ca-builder-scripts/configs/batch-of-certs.txt",
"content": "consumer,machine0.example.com\nproducer,machine1.example.com\nkafka,machine2.example.com\nzookeeper,machine3.example.com\n"
},
{
"path": "ca-builder-scripts/configs/batch-of-stores.txt",
"content": "consumer,machine0.example.com\nproducer,machine1.example.com\nkafka,machine2.example.com\nzookeeper,machine3.example.com\n"
},
{
"path": "ca-builder-scripts/configs/ca-config-vars",
"content": "DE\nBerlin\nBerlin\nConfluent Germany\n"
},
{
"path": "ca-builder-scripts/configs/ca.config",
"content": "[ ca ]\ndefault_ca = CA_default\n\n[ CA_default ]\n# Directory and file locations.\ndir = .\ncerts = $dir/certs\ncrl_dir = $dir/crl\nnew_certs_dir = $dir/newcerts\ndatabase = $dir/index.txt\nserial = $dir/serial\nRANDFILE = $dir/private/.rand\n\n# The root key and root certificate.\nprivate_key = $dir/private/ca.key.pem\ncertificate = $dir/certs/ca.cert.pem\n\n# For certificate revocation lists.\ncrlnumber = $dir/crlnumber\ncrl = $dir/crl/ca.crl.pem\ncrl_extensions = crl_ext\ndefault_crl_days = 30\n\n# SHA-1 is deprecated, so use SHA-2 instead.\ndefault_md = sha256\n\nname_opt = ca_default\ncert_opt = ca_default\ndefault_days = 375\npreserve = no\npolicy = policy_strict\n\n[ policy_strict ]\n# The root CA should only sign intermediate certificates that match.\n# See the POLICY FORMAT section of `man ca`.\ncountryName = match\nstateOrProvinceName = match\norganizationName = match\norganizationalUnitName = optional\ncommonName = supplied\nemailAddress = optional\n\n[ policy_loose ]\n# Allow the intermediate CA to sign a more diverse range of certificates.\n# See the POLICY FORMAT section of the `ca` man page.\ncountryName = optional\nstateOrProvinceName = optional\nlocalityName = optional\norganizationName = optional\norganizationalUnitName = optional\ncommonName = supplied\nemailAddress = optional\n\n[ req ]\n# Options for the `req` tool (`man req`).\ndefault_bits = 2048\ndistinguished_name = req_distinguished_name\nstring_mask = utf8only\n\n# SHA-1 is deprecated, so use SHA-2 instead.\ndefault_md = sha256\n\n# Extension to add when the -x509 option is used.\nx509_extensions = v3_ca\n\n[ req_distinguished_name ]\n# See .\ncountryName = Country Name (2 letter code)\nstateOrProvinceName = State or Province Name\nlocalityName = Locality Name\n0.organizationName = Organization Name\norganizationalUnitName = Organizational Unit Name\ncommonName = Common Name\nemailAddress = Email Address\n\n# Optionally, specify some defaults.\ncountryName_default = DE\nstateOrProvinceName_default = Berlin\nlocalityName_default = Berlin\n0.organizationName_default = Confluent Ltd\n#organizationalUnitName_default =\n#emailAddress_default =\n\n\n[ v3_ca ]\n# Extensions for a typical CA (`man x509v3_config`).\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid:always,issuer\nbasicConstraints = critical, CA:true\nkeyUsage = critical, digitalSignature, cRLSign, keyCertSign\n\n\n[ v3_intermediate_ca ]\n# Extensions for a typical intermediate CA (`man x509v3_config`).\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid:always,issuer\nbasicConstraints = critical, CA:true, pathlen:0\nkeyUsage = critical, digitalSignature, cRLSign, keyCertSign\n\n[ usr_cert ]\n# Extensions for client certificates (`man x509v3_config`).\nbasicConstraints = CA:FALSE\nnsCertType = client, email\nnsComment = \"OpenSSL Generated Client Certificate\"\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid,issuer\nkeyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment\nextendedKeyUsage = clientAuth, emailProtection\n\n[ server_cert ]\n# Extensions for server certificates (`man x509v3_config`).\nbasicConstraints = CA:FALSE\nnsCertType = server\nnsComment = \"OpenSSL Generated Server Certificate\"\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid,issuer:always\nkeyUsage = critical, digitalSignature, keyEncipherment\nextendedKeyUsage = serverAuth\n\n[ crl_ext ]\n# Extension for CRLs (`man x509v3_config`).\nauthorityKeyIdentifier=keyid:always\n\n[ ocsp ]\n# Extension for OCSP signing certificates (`man ocsp`).\nbasicConstraints = CA:FALSE\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid,issuer\nkeyUsage = critical, digitalSignature\nextendedKeyUsage = critical, OCSPSigning\n"
},
{
"path": "ca-builder-scripts/configs/intermediate-ca.config",
"content": "[defaults]\ncrl_url = http://httpd:80/crls.pem # CRL distribution point\n\n[ ca ]\n# `man ca`\ndefault_ca = CA_default\n\n[ CA_default ]\n# Directory and file locations.\ndir = intermediate/\ncerts = $dir/certs\ncrl_dir = $dir/crl\nnew_certs_dir = $dir/newcerts\ndatabase = $dir/index.txt\nserial = $dir/serial\nRANDFILE = $dir/private/.rand\n\n# The root key and root certificate.\nprivate_key = $dir/private/intermediate.key.pem\ncertificate = $dir/certs/intermediate.cert.pem\n\n# For certificate revocation lists.\ncrlnumber = $dir/crlnumber\ncrl = $dir/crl/intermediate.crl.pem\ncrl_extensions = crl_ext\ndefault_crl_days = 30\n\n\n# SHA-1 is deprecated, so use SHA-2 instead.\ndefault_md = sha256\n\nname_opt = ca_default\ncert_opt = ca_default\ndefault_days = 375\npreserve = no\npolicy = policy_loose\n\n[ policy_strict ]\n# The root CA should only sign intermediate certificates that match.\n# See the POLICY FORMAT section of `man ca`.\ncountryName = match\nstateOrProvinceName = match\norganizationName = match\norganizationalUnitName = optional\ncommonName = supplied\nemailAddress = optional\n\n[ policy_loose ]\n# Allow the intermediate CA to sign a more diverse range of certificates.\n# See the POLICY FORMAT section of the `ca` man page.\ncountryName = optional\nstateOrProvinceName = optional\nlocalityName = optional\norganizationName = optional\norganizationalUnitName = optional\ncommonName = supplied\nemailAddress = optional\n\n[ req ]\n# Options for the `req` tool (`man req`).\ndefault_bits = 2048\ndistinguished_name = req_distinguished_name\nstring_mask = utf8only\n\n# SHA-1 is deprecated, so use SHA-2 instead.\ndefault_md = sha256\n\n# Extension to add when the -x509 option is used.\nx509_extensions = v3_ca\n\n[ req_distinguished_name ]\n# See .\ncountryName = Country Name (2 letter code)\nstateOrProvinceName = State or Province Name\nlocalityName = Locality Name\n0.organizationName = Organization Name\norganizationalUnitName = Organizational Unit Name\ncommonName = Common Name\nemailAddress = Email Address\n\n# Optionally, specify some defaults.\ncountryName_default = DE\nstateOrProvinceName_default = Berlin\nlocalityName_default = Berlin\n0.organizationName_default = Confluent Ltd\norganizationalUnitName_default =\nemailAddress_default =\n\n[ v3_ca ]\n# Extensions for a typical CA (`man x509v3_config`).\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid:always,issuer\nbasicConstraints = critical, CA:true\nkeyUsage = critical, digitalSignature, cRLSign, keyCertSign\n\n[ v3_intermediate_ca ]\n# Extensions for a typical intermediate CA (`man x509v3_config`).\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid:always,issuer\nbasicConstraints = critical, CA:true, pathlen:0\nkeyUsage = critical, digitalSignature, cRLSign, keyCertSign\n\n[ usr_cert ]\n# Extensions for client certificates (`man x509v3_config`).\nbasicConstraints = CA:FALSE\nnsCertType = client, email\nnsComment = \"OpenSSL Generated Client Certificate\"\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid,issuer\nkeyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment\nextendedKeyUsage = clientAuth, emailProtection\n\n[ server_cert ]\n# Extensions for server certificates (`man x509v3_config`).\nbasicConstraints = CA:FALSE\nnsCertType = client, server\nnsComment = \"OpenSSL Generated Server Certificate\"\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid,issuer:always\nkeyUsage = critical, digitalSignature, keyEncipherment\nextendedKeyUsage = clientAuth, serverAuth\ncrlDistributionPoints = URI:http://httpd:80/crls.pem\nsubjectAltName = @alt_names\n\n[ crl_ext ]\n# Extension for CRLs (`man x509v3_config`).\nauthorityKeyIdentifier=keyid:always\n\n[ ocsp ]\n# Extension for OCSP signing certificates (`man ocsp`).\nbasicConstraints = CA:FALSE\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid,issuer\nkeyUsage = critical, digitalSignature\nextendedKeyUsage = critical, OCSPSigning\n"
},
{
"path": "ca-builder-scripts/create-crl.sh",
"content": "#!/usr/bin/env bash\n\n\nDEFAULT_PASSWORD=${1:-confluent}\n\nif [ -z \"${CA_ROOT_DIR+x}\" ];\nthen\nCA_ROOT_DIR='.'\nfi\n\nsource $CA_ROOT_DIR/utils/functions.sh\n\n(cd $CA_ROOT_DIR/ca; create_certificate_revokation_list )\n"
},
{
"path": "ca-builder-scripts/create-pair-certs.sh",
"content": "#!/usr/bin/env bash\n\n#HOSTNAME=\"www.example.com\"\n#EXTENSION=\"server_cert\" #usr_cert for client auth, server_cert for for backend\n\n#HOSTNAME=\"my.kafka.consumer\"\n#EXTENSION=\"usr_cert\"\nset -e\n\nHOSTNAME=$1\nMACHINE=${2:-\"\"}\nEXTENSION=${3:-server_cert}\nDEFAULT_PASSWORD=${4:-confluent}\n\necho \"Building a part of certificates for $HOSTNAME using $EXTENSION\"\n\nif [ -z \"${CA_ROOT_DIR+x}\" ];\nthen\nCA_ROOT_DIR='.'\nfi\n\nITERMEDIATE_CA_DIR=$CA_ROOT_DIR/ca/intermediate\n\nCERT_FILE=\"$ITERMEDIATE_CA_DIR/certs/$HOSTNAME.cert.pem\"\n\nif test -f \"$CERT_FILE\"; then\n RED='\\033[0;31m'\n NC='\\033[0m' # No Color\n printf \"${RED}Cert $CERT_FILE exist! exiting...${NC}\"\n exit 1\nfi\n\nsource $CA_ROOT_DIR/utils/functions.sh\n\n(cd $CA_ROOT_DIR; refresh_openssl_file \"$CA_ROOT_DIR\" \"$ITERMEDIATE_CA_DIR\" )\n(cd $CA_ROOT_DIR/ca; generate_final_certificate \"$MACHINE\" )\n"
},
{
"path": "ca-builder-scripts/del-cert.sh",
"content": "#!/usr/bin/env bash\n\nNAME=$1\n\nif [ -z \"${CA_ROOT_DIR+x}\" ];\nthen\nCA_ROOT_DIR='.'\nfi\n\necho \"Deleting CERT $NAME\"\n\nrm \"$CA_ROOT_DIR/ca/intermediate/private/$NAME.key.pem\"\nrm \"$CA_ROOT_DIR/ca/intermediate/certs/$NAME.cert.pem\"\nrm \"$CA_ROOT_DIR/ca/intermediate/csr/$NAME.csr.pem\"\n"
},
{
"path": "ca-builder-scripts/revoke-cert.sh",
"content": "#!/usr/bin/env bash\n\nCERT=$1\nDEFAULT_PASSWORD=${2:-confluent}\n\nif [ -z \"${CA_ROOT_DIR+x}\" ];\nthen\nCA_ROOT_DIR='.'\nfi\n\nsource $CA_ROOT_DIR/utils/functions.sh\n\n(cd $CA_ROOT_DIR/ca; revoke_cert $CERT )\n"
},
{
"path": "ca-builder-scripts/setup-ca-with-intermediate-ca.sh",
"content": "#!/usr/bin/env bash\n\n##\n# This script builds a Certificate Authority of the form:\n# Root CA -> intermediate CA\n#\n# In the CA_ROOT_DIR, this script will create the necessary directory strucures\n# and generate the certificates, all signed using the value provided as an\n# argument to this script, or confluent by default.\n##\n\nDEFAULT_PASSWORD=${1:-confluent}\nexport CA_ROOT_DIR=`pwd`\n\necho -e \"Building the CA root setup\\n\"\n\n./utils/build-ca.sh $DEFAULT_PASSWORD\n\necho -e \"Building the intemedite CA root setup:\\n\"\n\n./utils/build-intermediate-ca.sh $DEFAULT_PASSWORD\n"
},
{
"path": "ca-builder-scripts/support-scripts/build-ca.sh",
"content": "#!/usr/bin/expect\n\nproc slurp {file} {\n set fh [open $file r]\n set ret [read $fh]\n close $fh\n return $ret\n}\n\nset timeout 20\nset configslurp [slurp configs/ca-config-vars]\n\nset lines [split $configslurp \\n]\nset COUNTRY_NAME [lrange $lines 0 0]\nset STATE [lrange $lines 1 1]\nset LOCALITY [lrange $lines 2 2]\nset ORGANIZATION [lrange $lines 3 3]\n\neval spawn ./setup-ca-with-intermediate-ca.sh\n## Generating the data for the CA setup.\nexpect \"Country Name (2 letter code)\"\nsend \"$COUNTRY_NAME\\r\";\nexpect \"State or Province Name\"\nsend \"$STATE\\r\";\nexpect \"Locality Name\"\nsend \"$LOCALITY\\r\";\nexpect \"Organization Name\"\nsend \"$ORGANIZATION\\r\";\nexpect \"Organizational Unit Name\"\nsend \"\\r\";\nexpect \"Common Name\"\nsend \"CA\\r\";\nexpect \"Email Address\"\nsend \"\\r\";\n## Generating the data for the Intermediate setup.\nexpect \"Country Name (2 letter code)\"\nsend \"$COUNTRY_NAME\\r\";\nexpect \"State or Province Name\"\nsend \"$STATE\\r\";\nexpect \"Locality Name\"\nsend \"$LOCALITY\\r\";\nexpect \"Organization Name\"\nsend \"$ORGANIZATION\\r\";\nexpect \"Organizational Unit Name\"\nsend \"\\r\";\nexpect \"Common Name\"\nsend \"Intermediate-CA\\r\";\nexpect \"Email Address\"\nsend \"\\r\";\n# Sign the certificate and commit\nexpect \"Sign the certificate?\"\nsend \"y\\r\";\nexpect \"1 out of 1 certificate requests certified, commit?\"\nsend \"y\\r\";\ninteract\n"
},
{
"path": "ca-builder-scripts/support-scripts/create-cert.sh",
"content": "#!/usr/bin/expect -f\n\nproc slurp {file} {\n set fh [open $file r]\n set ret [read $fh]\n close $fh\n return $ret\n}\n\nproc create_certs {cert_name,machine} {\n eval spawn ./create-pair-certs.sh $cert_name $machine\n}\n\nset timeout 20\nset configslurp [slurp configs/ca-config-vars]\n\nset lines [split $configslurp \\n]\nset COUNTRY_NAME [lrange $lines 0 0]\nset STATE [lrange $lines 1 1]\nset LOCALITY [lrange $lines 2 2]\nset ORGANIZATION [lrange $lines 3 3]\n\nset cert_name [lindex $argv 0]\nset machine [lrange $argv 1 end]\n\nspawn ./create-pair-certs.sh $cert_name $machine\n\n## Generating the data for the CA setup.\nexpect \"Country Name (2 letter code)\"\nsend \"$COUNTRY_NAME\\r\";\nexpect \"State or Province Name\"\nsend \"$STATE\\r\";\nexpect \"Locality Name\"\nsend \"$LOCALITY\\r\";\nexpect \"Organization Name\"\nsend \"$ORGANIZATION\\r\";\nexpect \"Organizational Unit Name\"\nsend \"\\r\";\nexpect \"Common Name\"\nsend \"$cert_name\\r\";\nexpect \"Email Address\"\nsend \"\\r\";\n# Sign the certificate and commit\nexpect \"Sign the certificate?\"\nsend \"y\\r\";\nexpect \"1 out of 1 certificate requests certified, commit\"\nsend \"y\\r\";\ninteract\n"
},
{
"path": "ca-builder-scripts/utils/build-ca.sh",
"content": "#!/usr/bin/env bash\n\nDEFAULT_PASSWORD=${1:-confluent}\n\nif [ -z \"${CA_ROOT_DIR+x}\" ];\nthen\nCA_ROOT_DIR='.'\nfi\n\nsource $CA_ROOT_DIR/utils/functions.sh\n\nmkdir $CA_ROOT_DIR/ca;\n\nsetup_ca_dir_structure \"$CA_ROOT_DIR/ca\"\n\ncp $CA_ROOT_DIR/configs/ca.config $CA_ROOT_DIR/ca/openssl.cnf\n\n(cd $CA_ROOT_DIR/ca; generate_ca_keys_and_certs )\n\n## Verify the CA certificate\nopenssl x509 -noout -text -in $CA_ROOT_DIR/ca/certs/ca.cert.pem\n"
},
{
"path": "ca-builder-scripts/utils/build-intermediate-ca.sh",
"content": "#!/usr/bin/env bash\n\nDEFAULT_PASSWORD=${1:-confluent}\n\nif [ -z \"${CA_ROOT_DIR+x}\" ];\nthen\nCA_ROOT_DIR='.'\nfi\nITERMEDIATE_CA_DIR=$CA_ROOT_DIR/ca/intermediate\n\nsource $CA_ROOT_DIR/utils/functions.sh\n\nmkdir -p $ITERMEDIATE_CA_DIR\n\nsetup_intermediate_ca_dir_structure $ITERMEDIATE_CA_DIR\n\ncp $CA_ROOT_DIR/configs/intermediate-ca.config $ITERMEDIATE_CA_DIR/openssl.cnf\n\n(cd $ITERMEDIATE_CA_DIR; generate_intermediate_keys_and_certs)\n\n(cd $CA_ROOT_DIR/ca; sign_intermediate_cert_authority; verify_generate_intermediate_ca)\n(cd $CA_ROOT_DIR/ca; create_ca_chain)\n"
},
{
"path": "ca-builder-scripts/utils/functions.sh",
"content": "#!/usr/bin/env bash\n\ngenerate_ca_keys_and_certs () {\n\nopenssl genrsa -aes256 -passout pass:$DEFAULT_PASSWORD -out private/ca.key.pem 4096\nchmod 400 private/ca.key.pem\n\nopenssl req -config openssl.cnf \\\n -key private/ca.key.pem \\\n -new -x509 -days 7300 -sha256 -extensions v3_ca \\\n -passin pass:$DEFAULT_PASSWORD -passout pass:$DEFAULT_PASSWORD \\\n -out certs/ca.cert.pem\n\n chmod 444 certs/ca.cert.pem\n}\n\nsetup_ca_dir_structure() {\n mkdir -p $1/certs $1/crl $1/newcerts $1/private\n chmod 700 $1/private\n touch $1/index.txt\n echo 1000 > $1/serial\n}\n\nsetup_intermediate_ca_dir_structure() {\n setup_ca_dir_structure $1\n mkdir -p $1/csr\n echo 1000 > $1/crlnumber\n}\n\n\ngenerate_intermediate_keys_and_certs () {\n openssl genrsa -aes256 -passout pass:$DEFAULT_PASSWORD -out private/intermediate.key.pem 4096\n chmod 400 private/intermediate.key.pem\n\n openssl req -config openssl.cnf -new -sha256 \\\n -passin pass:$DEFAULT_PASSWORD -passout pass:$DEFAULT_PASSWORD \\\n -key private/intermediate.key.pem \\\n -out csr/intermediate.csr.pem\n}\n\nsign_intermediate_cert_authority () {\n # signature\n openssl ca -config openssl.cnf -extensions v3_intermediate_ca \\\n -days 3650 -notext -md sha256 \\\n -in intermediate/csr/intermediate.csr.pem \\\n -passin pass:$DEFAULT_PASSWORD \\\n -out intermediate/certs/intermediate.cert.pem\n chmod 444 intermediate/certs/intermediate.cert.pem\n}\n\nverify_generate_intermediate_ca () {\n # verification\n openssl x509 -noout -text -in intermediate/certs/intermediate.cert.pem\n openssl verify -CAfile certs/ca.cert.pem intermediate/certs/intermediate.cert.pem\n}\n\ncreate_ca_chain () {\n # create the CA chain\n cat intermediate/certs/intermediate.cert.pem certs/ca.cert.pem > intermediate/certs/ca-chain.cert.pem\n chmod 444 intermediate/certs/ca-chain.cert.pem\n}\n\nrefresh_openssl_file() {\n ca_root_dir=$1\n intermediate_dir=$2\n cp $ca_root_dir/configs/intermediate-ca.config $intermediate_dir/openssl.cnf\n}\n\ngenerate_final_certificate () {\n alt_name=$1\n echo \"$DEFAULT_PASSWORD\"\n # create a private key\n openssl genrsa -aes256 -passout pass:$DEFAULT_PASSWORD -out intermediate/private/$HOSTNAME.key.pem 2048\n chmod 400 intermediate/private/$HOSTNAME.key.pem\n\n echo -e \"\" >> intermediate/openssl.cnf\n echo -e \"[ alt_names ]\" >> intermediate/openssl.cnf\n echo -e \"DNS.1=localhost\" >> intermediate/openssl.cnf\n echo -e \"DNS.2=$alt_name\" >> intermediate/openssl.cnf\n\n # create a csr\n openssl req -config intermediate/openssl.cnf \\\n -passin pass:confluent -passout pass:$DEFAULT_PASSWORD \\\n -key intermediate/private/$HOSTNAME.key.pem \\\n -new -sha256 -out intermediate/csr/$HOSTNAME.csr.pem\n\n # create the cert\n openssl ca -config intermediate/openssl.cnf -extensions $EXTENSION -days 375 -notext -md sha256 \\\n -in intermediate/csr/$HOSTNAME.csr.pem \\\n -passin pass:$DEFAULT_PASSWORD \\\n -out intermediate/certs/$HOSTNAME.cert.pem\n\n chmod 444 intermediate/certs/$HOSTNAME.cert.pem\n\n # verify the cert\n openssl x509 -noout -text -in intermediate/certs/$HOSTNAME.cert.pem\n\n # verify the chain trust\n openssl verify -CAfile intermediate/certs/ca-chain.cert.pem intermediate/certs/$HOSTNAME.cert.pem\n}\n\ncreate_certificate_revokation_list () {\n openssl ca -config intermediate/openssl.cnf -gencrl \\\n -passin pass:$DEFAULT_PASSWORD \\\n -out intermediate/crl/intermediate.crl.pem\n}\n\nrevoke_cert () {\n openssl ca -config intermediate/openssl.cnf -passin pass:$DEFAULT_PASSWORD -revoke \"intermediate/certs/$1.cert.pem\"\n}\n"
},
{
"path": "delegation_tokens/.gitignore",
"content": "certs/\n"
},
{
"path": "delegation_tokens/ca.cnf",
"content": "[ policy_match ]\ncountryName = match\nstateOrProvinceName = match\norganizationName = match\norganizationalUnitName = optional\ncommonName = supplied\nemailAddress = optional\n\n[ req ]\nprompt = no\ndistinguished_name = dn\ndefault_md = sha256\ndefault_bits = 4096\nx509_extensions = v3_ca\n\n[ dn ]\ncountryName = UK\norganizationName = Confluent\nlocalityName = London\ncommonName = kafka.confluent.local\n\n[ v3_ca ]\nsubjectKeyIdentifier=hash\nbasicConstraints = critical,CA:true\nauthorityKeyIdentifier=keyid:always,issuer:always\nkeyUsage = critical,keyCertSign,cRLSign\n"
},
{
"path": "delegation_tokens/client.cnf",
"content": "[req]\nprompt = no\ndistinguished_name = dn\ndefault_md = sha256\ndefault_bits = 4096\nreq_extensions = v3_req\n\n[ dn ]\ncountryName = UK\norganizationName = Confluent\nlocalityName = London\ncommonName=kafka.confluent.local\n\n[ v3_ca ]\nsubjectKeyIdentifier=hash\nbasicConstraints = critical,CA:true\nauthorityKeyIdentifier=keyid:always,issuer:always\nkeyUsage = critical,keyCertSign,cRLSign\n\n[ v3_req ]\nsubjectKeyIdentifier = hash\nbasicConstraints = CA:FALSE\nnsComment = \"OpenSSL Generated Certificate\"\nkeyUsage = critical, digitalSignature, keyEncipherment\nextendedKeyUsage = clientAuth\nsubjectAltName = @alt_names\n\n[ alt_names ]\nDNS.1=kafka.confluent.local\n"
},
{
"path": "delegation_tokens/docker-compose.yml",
"content": "version: '3'\nservices:\n zookeeper:\n build: zookeeper/\n container_name: zookeeper\n hostname: zookeeper\n domainname: confluent.local\n restart: on-failure\n volumes:\n - ./certs/:/var/lib/secret\n environment:\n - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/zookeeper_server_jaas.conf\n networks:\n default:\n aliases:\n - zookeeper.confluent.local\n\n\n kafka:\n build: kafka/\n container_name: kafka\n hostname: kafka\n domainname: confluent.local\n depends_on: \n - zookeeper\n restart: on-failure\n volumes:\n - ./certs/:/var/lib/secret\n environment:\n - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf\n networks:\n default:\n aliases:\n - kafka.confluent.local\n ports:\n - \"9093:9093\"\n\nvolumes:\n secret: {}\n\nnetworks:\n default:\n"
},
{
"path": "delegation_tokens/kafka/Dockerfile",
"content": "FROM centos\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.4/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-1.8.0-openjdk\nRUN yum install -y confluent-platform-2.12\n\n# 3. Configure Kafka \nCOPY server.properties /etc/kafka/server.properties\nCOPY kafka_server_jaas.conf /etc/kafka/kafka_server_jaas.conf\nCOPY consumer.properties /etc/kafka/consumer.properties\nCOPY create_client_properties.sh /etc/kafka/create_client_properties.sh\n\nEXPOSE 9093\n\nCMD kafka-server-start /etc/kafka/server.properties\n"
},
{
"path": "delegation_tokens/kafka/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.4/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.4\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n"
},
{
"path": "delegation_tokens/kafka/consumer.properties",
"content": "sasl.mechanism=SCRAM-SHA-256\n# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT\nsecurity.protocol=SASL_SSL\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"kafka\" \\\n password=\"kafka\";\nssl.truststore.location=/var/lib/secret/truststore.jks\nssl.truststore.password=test1234\nssl.keystore.location=/var/lib/secret/client.keystore.jks\nssl.keystore.password=test1234\n"
},
{
"path": "delegation_tokens/kafka/create_client_properties.sh",
"content": "#!/bin/bash\n\nset -e\nset -u\n\nRESPONSE=$(kafka-delegation-tokens \\\n --bootstrap-server kafka.confluent.local:9093 \\\n --create \\\n --command-config /etc/kafka/consumer.properties \\\n --max-life-time-period -1 | tail -1)\n\nTOKENID=$(echo $RESPONSE | cut -d \" \" -f1)\nHMAC=$(echo $RESPONSE | cut -d \" \" -f2)\n\necho \"Received token id: $TOKENID\"\necho \"Received message authentication code: $HMAC\"\n\necho 'sasl.mechanism=SCRAM-SHA-256\n# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT\nsecurity.protocol=SASL_SSL\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"'$TOKENID'\" \\\n password=\"'$HMAC'\" \\\n tokenauth=\"true\";\nssl.truststore.location=/var/lib/secret/truststore.jks\nssl.truststore.password=test1234\nssl.keystore.location=/var/lib/secret/client.keystore.jks\nssl.keystore.password=test1234' > /tmp/delegation_token_client.properties\n\n"
},
{
"path": "delegation_tokens/kafka/kafka_server_jaas.conf",
"content": "// Username and password are used by the broker to initiate connections to other brokers\n// admin is another user allowed to connect to the broker.\n\nKafkaServer {\n org.apache.kafka.common.security.scram.ScramLoginModule required\n username=\"kafka\"\n password=\"kafka\"\n user_admin=\"admin\";\n};\n\n// The client section is used by kafka to connect to zookeeper.\n// This must match the zookeeper jaas configuration.\nClient {\n org.apache.zookeeper.server.auth.DigestLoginModule required\n username=\"kafka\"\n password=\"kafka\";\n};\n"
},
{
"path": "delegation_tokens/kafka/server.properties",
"content": "############################# Server Basics #############################\nbroker.id=0\nlisteners=SASL_SSL://kafka.confluent.local:9093\nadvertised.listeners=SASL_SSL://kafka.confluent.local:9093\nlog.dirs=/var/lib/kafka\noffsets.topic.replication.factor=1\ntransaction.state.log.replication.factor=1\ntransaction.state.log.min.isr=1\nzookeeper.connect=zookeeper.confluent.local:2181\n\n# TLS Configuration\nsecurity.inter.broker.protocol=SASL_SSL\nssl.truststore.location=/var/lib/secret/truststore.jks\nssl.truststore.password=test1234\nssl.keystore.location=/var/lib/secret/server.keystore.jks\nssl.keystore.password=test1234\nssl.client.auth=required\nauthorizer.class.name=kafka.security.auth.SimpleAclAuthorizer\ndelegation.token.master.key=foo\nsasl.enabled.mechanisms=SCRAM-SHA-256\nsasl.mechanism.inter.broker.protocol=SCRAM-SHA-256\nsuper.users=User:kafka\n"
},
{
"path": "delegation_tokens/server.cnf",
"content": "[req]\nprompt = no\ndistinguished_name = dn\ndefault_md = sha256\ndefault_bits = 4096\nreq_extensions = v3_req\n\n[ dn ]\ncountryName = UK\norganizationName = Confluent\nlocalityName = London\ncommonName=kafka.confluent.local\n\n[ v3_ca ]\nsubjectKeyIdentifier=hash\nbasicConstraints = critical,CA:true\nauthorityKeyIdentifier=keyid:always,issuer:always\nkeyUsage = critical,keyCertSign,cRLSign\n\n[ v3_req ]\nsubjectKeyIdentifier = hash\nbasicConstraints = CA:FALSE\nnsComment = \"OpenSSL Generated Certificate\"\nkeyUsage = critical, digitalSignature, keyEncipherment\nextendedKeyUsage = serverAuth, clientAuth\nsubjectAltName = @alt_names\n\n[ alt_names ]\nDNS.1=kafka.confluent.local\n"
},
{
"path": "delegation_tokens/up",
"content": "#!/bin/sh\nset -e\n\n# Creating TLS CA, Certificates and keystore / truststore\nrm -rf certs \nmkdir -p certs\n# Generate CA certificates\nopenssl req -new -nodes -x509 -days 3650 -newkey rsa:2048 -keyout certs/ca.key -out certs/ca.crt -config ca.cnf\ncat certs/ca.crt certs/ca.key > certs/ca.pem\n\n# Generate kafka server certificates\nopenssl req -new -newkey rsa:2048 -keyout certs/server.key -out certs/server.csr -config server.cnf -nodes\nopenssl x509 -req -days 3650 -in certs/server.csr -CA certs/ca.crt -CAkey certs/ca.key -CAcreateserial -out certs/server.crt -extfile server.cnf -extensions v3_req\nopenssl pkcs12 -export -in certs/server.crt -inkey certs/server.key -chain -CAfile certs/ca.pem -name \"kafka.confluent.local\" -out certs/server.p12 -password pass:test1234\n\n# Generate client certificates\nopenssl req -new -newkey rsa:2048 -keyout certs/client.key -out certs/client.csr -config client.cnf -nodes\nopenssl x509 -req -days 3650 -in certs/client.csr -CA certs/ca.crt -CAkey certs/ca.key -CAcreateserial -out certs/client.crt -extfile client.cnf -extensions v3_req\nopenssl pkcs12 -export -in certs/client.crt -inkey certs/client.key -chain -CAfile certs/ca.pem -name \"kafka.confluent.local\" -out certs/client.p12 -password pass:test1234\n\n# Import server certificate to keystore and CA to truststore\nkeytool -importkeystore -deststorepass test1234 -destkeystore certs/server.keystore.jks \\\n -srckeystore certs/server.p12 \\\n -deststoretype PKCS12 \\\n -srcstoretype PKCS12 \\\n -noprompt \\\n -srcstorepass test1234\n\nkeytool -importkeystore -deststorepass test1234 -destkeystore certs/client.keystore.jks \\\n -srckeystore certs/client.p12 \\\n -deststoretype PKCS12 \\\n -srcstoretype PKCS12 \\\n -noprompt \\\n -srcstorepass test1234\n\nkeytool -keystore certs/truststore.jks -alias CARoot -import -file certs/ca.crt -storepass test1234 -noprompt -storetype PKCS12 \n\n# Starting docker-compose services\ndocker-compose up -d --build\n\ndocker-compose exec kafka kafka-configs \\\n --zookeeper zookeeper.confluent.local:2181 \\\n --alter --add-config 'SCRAM-SHA-256=[password=kafka],SCRAM-SHA-512=[password=kafka]' \\\n --entity-type users --entity-name kafka\ndocker-compose exec kafka kafka-acls \\\n --authorizer kafka.security.auth.SimpleAclAuthorizer \\\n --authorizer-properties zookeeper.connect=zookeeper.confluent.local:2181 \\\n --add --allow-principal User:kafka --operation All --topic test\n\ndocker exec kafka /etc/kafka/create_client_properties.sh\n\necho \"Example configuration to access kafka:\"\necho \"-> docker-compose exec kafka kafka-console-producer --broker-list kafka.confluent.local:9093 --topic test --producer.config /tmp/delegation_token_client.properties\"\necho \"-> docker-compose exec kafka kafka-console-consumer --bootstrap-server kafka.confluent.local:9093 --topic test --consumer.config /tmp/delegation_token_client.properties --from-beginning\"\n"
},
{
"path": "delegation_tokens/zookeeper/Dockerfile",
"content": "FROM centos\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.4/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-1.8.0-openjdk\nRUN yum install -y confluent-platform-2.12\n\n# 3. Configure zookeeper\nCOPY zookeeper.properties /etc/kafka/zookeeper.properties\nCOPY zookeeper.sasl.jaas.config /etc/kafka/zookeeper_server_jaas.conf\n\nEXPOSE 2181\n\nCMD zookeeper-server-start /etc/kafka/zookeeper.properties \n"
},
{
"path": "delegation_tokens/zookeeper/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.4/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.4\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n"
},
{
"path": "delegation_tokens/zookeeper/zookeeper.properties",
"content": "dataDir=/var/lib/zookeeper\nclientPort=2181\nmaxClientCnxns=0\nauthProvider.1 = org.apache.zookeeper.server.auth.SASLAuthenticationProvider\nrequireClientAuthScheme=sasl\n\n"
},
{
"path": "delegation_tokens/zookeeper/zookeeper.sasl.jaas.config",
"content": "Server {\n org.apache.zookeeper.server.auth.DigestLoginModule required\n user_kafka=\"kafka\";\n};\n"
},
{
"path": "kafka-connect-mtls/.gitignore",
"content": "connect/secrets/client-*.pem\nconnect/secrets/client.p12\n"
},
{
"path": "kafka-connect-mtls/README.md",
"content": "# Kafka Connect REST api ssl client auth\n\nOne of the common question regarding security on Kafka Connect REST api is how to prevent unwanted access.\nThis playbook show one of the possible methods currently possible (as of November 2019) using the SSL mTLS feature.\n\n## Requirements\n\nTo be able to execute this playbook you require:\n\n* Docker (19.03 or later)\n* Docker compose (1.24.1 or later)\n* curl\n\n## Bootstrap the playbook\n\nThe playbook bootstrap can be done by executing the ```./up``` script.\n\n### Prepared TLS certificates and keystores\n\nA set of prepared TLS certificates and keystores are available within the _connect/secrets_ directory.\nMost relevant ones are:\n\n* _certificate.p12_: TLS certificate to verify the failure of mTLS (this is a self sign certificate)\n* _rest-client.p12_: TLS certificate to verify the positive verification using mTLS (this cert is sign by the same CA as the server identity)\n* _server.keystore_ and _server.truststore_: keystores prepared for the Kafka Connect REST server identity.\n\nAll this certs has been created with the ca-builder-scripts.\n\n## Verify the connectivity\n\nTo verify the connectivity there is a prepared script ```check-ssl-client-auth.sh```.\nThis script uses curl to verify a success and a failure authentication using mTLS\n"
},
{
"path": "kafka-connect-mtls/check-ssl-client-auth.sh",
"content": "#!/usr/bin/env bash\n\nverify_ok_ssl_client_auth () {\n cp -f ../ca-builder-scripts/ca/intermediate/private/$1.key.pem connect/secrets/$1.key.pem\n cp -f ../ca-builder-scripts/ca/intermediate/certs/ca-chain.cert.pem connect/secrets/ca-chain.cert.pem\n cp -f ../ca-builder-scripts/ca/intermediate/certs/$1.cert.pem connect/secrets/$1.cert.pem\n curl --key connect/secrets/$1.key.pem --cacert connect/secrets/ca-chain.cert.pem --cert connect/secrets/$1.cert.pem:confluent https://localhost:18083\n}\n\nverify_ko_ssl_client_auth() {\n mkdir connect/certs\n openssl req -new -nodes -x509 -days 3650 -newkey rsa:2048 -keyout connect/certs/ca.key -out connect/certs/ca.crt -config connect/config/ca.cnf\n cat connect/certs/ca.crt connect/certs/ca.key > connect/certs/ca.pem\n\n openssl req -new -newkey rsa:2048 -keyout connect/certs/client.key -out connect/certs/client.csr -config connect/config/client.cnf -nodes\n openssl x509 -req -days 3650 -in connect/certs/client.csr -CA connect/certs/ca.crt -CAkey connect/certs/ca.key -CAcreateserial -out connect/certs/client.crt -extfile connect/config/client.cnf -extensions v3_req\n openssl pkcs12 -export -in connect/certs/client.crt -inkey connect/certs/client.key -chain -CAfile connect/certs/ca.pem -name \"connect\" -out connect/certs/client.p12 -password pass:confluent\n\n cp connect/certs/client.p12 connect/secrets/client.p12\n rm -rf connect/certs\n\n openssl pkcs12 -in connect/secrets/client.p12 -out connect/secrets/client-ca.pem -cacerts -nokeys -passin pass:confluent -passout pass:confluent\n openssl pkcs12 -in connect/secrets/client.p12 -out connect/secrets/client-client.pem -clcerts -nokeys -passin pass:confluent -passout pass:confluent\n openssl pkcs12 -in connect/secrets/client.p12 -out connect/secrets/client-key.pem -nocerts -passin pass:confluent -passout pass:confluent\n\n curl --insecure --key connect/secrets/client-key.pem --cacert connect/secrets/client-ca.pem --cert connect/secrets/client-client.pem:confluent https://localhost:18083\n}\n\n\necho \"Check SSL client auth with an unknown certificate\"\nverify_ko_ssl_client_auth\n\necho \"\"\necho \"\"\n\necho \"Check SSL client auth with a valid client\"\nverify_ok_ssl_client_auth \"connect\"\n"
},
{
"path": "kafka-connect-mtls/connect/config/ca.cnf",
"content": "[ policy_match ]\ncountryName = match\nstateOrProvinceName = match\norganizationName = match\norganizationalUnitName = optional\ncommonName = supplied\nemailAddress = optional\n\n[ req ]\nprompt = no\ndistinguished_name = dn\ndefault_md = sha256\ndefault_bits = 4096\nx509_extensions = v3_ca\n\n[ dn ]\ncountryName = DE\norganizationName = Confluent\nlocalityName = Berlin\ncommonName = connect.confluent.local\n\n[ v3_ca ]\nsubjectKeyIdentifier=hash\nbasicConstraints = critical,CA:true\nauthorityKeyIdentifier=keyid:always,issuer:always\nkeyUsage = critical,keyCertSign,cRLSign\n"
},
{
"path": "kafka-connect-mtls/connect/config/client.cnf",
"content": "[req]\nprompt = no\ndistinguished_name = dn\ndefault_md = sha256\ndefault_bits = 4096\nreq_extensions = v3_req\n\n[ dn ]\ncountryName = UK\norganizationName = Confluent\nlocalityName = London\ncommonName=connect.client\n\n[ v3_ca ]\nsubjectKeyIdentifier=hash\nbasicConstraints = critical,CA:true\nauthorityKeyIdentifier=keyid:always,issuer:always\nkeyUsage = critical,keyCertSign,cRLSign\n\n[ v3_req ]\nsubjectKeyIdentifier = hash\nbasicConstraints = CA:FALSE\nnsComment = \"OpenSSL Generated Certificate\"\nkeyUsage = critical, digitalSignature, keyEncipherment\nextendedKeyUsage = clientAuth\nsubjectAltName = @alt_names\n\n[ alt_names ]\nDNS.1=localhost\n"
},
{
"path": "kafka-connect-mtls/connect/secrets/ca-chain.cert.pem",
"content": "-----BEGIN CERTIFICATE-----\nMIIF4TCCA8mgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwgYExCzAJBgNVBAYTAkRF\nMQ8wDQYDVQQIDAZCZXJsaW4xDzANBgNVBAcMBkJlcmxpbjEWMBQGA1UECgwNQ29u\nZmx1ZW50IEx0ZDELMAkGA1UECwwCUFMxCzAJBgNVBAMMAmNhMR4wHAYJKoZIhvcN\nAQkBFg9jYUBjb25mbHVlbnQuaW8wHhcNMTkxMTEyMTExMTMxWhcNMjkxMTA5MTEx\nMTMxWjB9MQswCQYDVQQGEwJERTEPMA0GA1UECAwGQmVybGluMRYwFAYDVQQKDA1D\nb25mbHVlbnQgTHRkMQswCQYDVQQLDAJQUzEYMBYGA1UEAwwPSW50ZXJtZWRpYXRl\nLUNBMR4wHAYJKoZIhvcNAQkBFg9jYUBjb25mbHVlbnQuaW8wggIiMA0GCSqGSIb3\nDQEBAQUAA4ICDwAwggIKAoICAQDMTnQGumpx9Byuq+z1APGFFp3lYDIY8rbScRkk\ndXQ1tukad9aOtQtGWfShOeQ7wndKvdYzcPN+AnGoszjd/gVeRDiXo0rEMGYgyglW\nvC1C88LWy/Wg26mgGTkMpeUfgN25lpfOUlzh0bVtPfcw154DXd5HZiHOiJ6CXytx\nbf+8M99SkQ0+X6MH2EBXMbOJBGGzZycMON7ONhdGhBrmClMHPMomMnprdL/W2TKa\n6SGXpxh3lLGYREqBSUlP6Owt0SMf/V5enbNrtllzmliBZbiUraGLohKDz8c8rcCX\nzz3zoBGkEGAJEFbXqoMa7OcVb2InWJTkfN2E4Z2akU6+9u6pvQ+q2e3KxHR42ure\nP4ilI6wzJ43UaPyFPzFpRd16akvIEDkdACU1FmlajW69Haye7Ey1HoI2tiJHDPAN\nJJbWUl59FS2DeFt7KvlBHdy81Z3D/G9QjSkOGCItabQLF8Jum2d5qMdekb3X8fTi\n2SPeDcgPyRat+6gKqsINzM6ChnXA0CFMbSGN6XpxARcxd95HR9zIStGdVKcPtnaN\nQ8o3/ehTNC8DVuT2LLlSwdAAElgZ/EHtfW7rrOBDgIMV8946m5r2tmf/K/A1vEUU\n4+QD66IX4nleK+OGuUexF1xv2UTzGhmk/VJsvx3AjPQgQLGqT2aRm6I6NEeWK/CY\njB597QIDAQABo2YwZDAdBgNVHQ4EFgQUJscsGzI7qLGpXsGdjRsvE6NXG0EwHwYD\nVR0jBBgwFoAU1K1JidhNRLmWIeRKobCfr8QhHQUwEgYDVR0TAQH/BAgwBgEB/wIB\nADAOBgNVHQ8BAf8EBAMCAYYwDQYJKoZIhvcNAQELBQADggIBAMCpI/ojEZc28fYF\nVbM9XYfPY6M8WRU39SSPZO5dknVmP16r6TInD2ZECfW/5XkqXPtvj3wcvXJ4/tLN\niS0DIzXbUlsO9sedDSCcAq251c2MIfROKj3E93X66PAVSn1SWVRgqlyD62G3Nv2E\nZ+V6vnE6oEXdrIucM4dUdvoz9Qnr0pZCchMYyFVdydAfvHu7wJ8PqVGgNCSgStuI\naQQUiKLFSnWOr+EJ8rZ8eGpgCWbeQ34ww8SlyjEPbkXAblsv0rH/R17GITIkFa69\nSHjOChvpOWcvZ0SLwisDShVBlRcO2ypwfo5avQCYBmj27D2U7htUmn+QwhntOSAB\nsAGUGeeoBgCKI7uNBuP3lXeaSXNWqsNXm7wSqsNJAPGVQzgA7kM1yjzdawCuAEzR\nJdFp0Y5juUaZ1FL6xnpvzSpBYBBs2o9CwtuBkBAnuG88AUwC7JR4URb0qb7/DbS0\nefbFPvUDohMAfYgUPXu1FemIuRicN99JoQLArEutKteGw4tX6W5oGckb6iBeQMe/\nC+Aw+RLEOyVwbfeF6lVfn4iZfn8A0SCqYRD+vStgKrb7LxPPl8/vTpgoiDH3qulD\nmV1vqeT3ESFvEnEXVd/QozDTn/AZsna3C2cp902GPEH3vV+h3ECRDZoaxuSdSzlg\n0d/kGEPjjbgBDD5IJ6mNgsiIBfiX\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIF6jCCA9KgAwIBAgIJAIopqcNHix7tMA0GCSqGSIb3DQEBCwUAMIGBMQswCQYD\nVQQGEwJERTEPMA0GA1UECAwGQmVybGluMQ8wDQYDVQQHDAZCZXJsaW4xFjAUBgNV\nBAoMDUNvbmZsdWVudCBMdGQxCzAJBgNVBAsMAlBTMQswCQYDVQQDDAJjYTEeMBwG\nCSqGSIb3DQEJARYPY2FAY29uZmx1ZW50LmlvMB4XDTE5MTExMjExMTA1OVoXDTM5\nMTEwNzExMTA1OVowgYExCzAJBgNVBAYTAkRFMQ8wDQYDVQQIDAZCZXJsaW4xDzAN\nBgNVBAcMBkJlcmxpbjEWMBQGA1UECgwNQ29uZmx1ZW50IEx0ZDELMAkGA1UECwwC\nUFMxCzAJBgNVBAMMAmNhMR4wHAYJKoZIhvcNAQkBFg9jYUBjb25mbHVlbnQuaW8w\nggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDOZyDRn9Zmv8WqhR6a8Sr5\nGXa3QO4yNB5F9PSAe4xXPOkDj14Om0cS/oXYfy/uJyVb3zxPMcF4Mz/WSQTATh2Y\nIunkYWqsrDZbgMG2ERdmSKvALKTd8mVFpCSzrhAvv8lZcRX0/jegjyMo2fFjRHfb\ndtwE341Ywgvf5gm0HH6UzbPIY9Xd/PmHJ1kRb+wJnRxs7T00l0pXZMYafs9tbJPv\n8tfrYjIM5PLxLAtlMZ1yLR1Ay8QMm5q4ZtTOQjlGOovWpPHHEhM3RealpYk1dC84\nngDpTI5zpf/Q0o4cn8aOACiy7fmm1dNMBR0Yu1JSLINxU+P0g3fNZu1PQjF13crm\nhGkcNAm6k0AMYwspNLthsPdOTgHfnv6/L/VkqGIBYU+gd0G2ZjbgpEHesQM91A42\njBneY1uFhcTjmAmx+MV63BWutFrkl1ErG4raYNIdUtGlsGhJ71AmOtGj3A8YyQAm\nLuzjZTQmkyP0saDa0wpsI7+lSZ5K5cJSQBMUf0bOQPspMmHfsO7H//RbmxdCiIHx\ng0fCfKX3oyungProDglptDYaNwE8oF0B1vl28cwlOJm7N9DVUsDc1bZwnfdquYD7\nq3rBkjZHyqmnsSVAOHvkToaHkkS7x+pFz/mPj/2Yo6h4Ujgrk4GO+vqQszk2Rp4Y\n58WbQ4Nq1qyrVIF0FgYjFwIDAQABo2MwYTAdBgNVHQ4EFgQU1K1JidhNRLmWIeRK\nobCfr8QhHQUwHwYDVR0jBBgwFoAU1K1JidhNRLmWIeRKobCfr8QhHQUwDwYDVR0T\nAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAYYwDQYJKoZIhvcNAQELBQADggIBABmT\nrHBFhz55quEG+/h3CiMKXpdld3cf27zgO2liEw/F3Xu04QaSpwsVZEOhuWjmzUA/\nlSfrbWeV/jNQz0Gs/rDdMUEDbNN+vsC4+u4xbAe9OYLsRJ5v1/PB7P/JxfhnRQLr\n3vQSUm7oEkhVxmTO0mFxpbrkcglAgA0siSWI2Kj0qTNeSxOW9PKOQuPqH9bTHRBt\nKoiR5CAealT7EoWUhZaNgLLOJ0fsIlPbXfAeat2BVRZAYQFQOJIOouZES49Yc+qF\nr6T7AVysk7pRilQWVovBOqSKzHTOKBKTJvbO23lYAOREc+UM694dPh480i6TcGEI\n4iYyC2/GCK4aUCCE0WTTqSErL0fNmjVz68lpuBjUS06+dGy2p6bW8fso6ttqAry+\njbEq6Ir+P9vZbvHOPc3JyaBKGyTRwA4jcrXQ5KD5LJwBd+9/h3elciNsJtd8mHdE\n2T3IqqsXdqN7gyaqwNQsk59iSkCr3cJGqvpsdVPXseadv9rcV5hxAjvWa4mLRiTL\nCxhs4CuaXz9kyss9oM1kWD0Dz3sIGwxtOcSREIqc8vclztjnpPMdly2lsK16hyfs\nq2P7yFurz6ruIxLR/f8SEalQ/Eqn8kOgwwHNuhaMHsaeS3o3Guwr+KHRm0jGNO/N\nZSm2YS9n3ghEYhUqfzuJCWuaeFELeOBb5wUx01Va\n-----END CERTIFICATE-----\n"
},
{
"path": "kafka-connect-mtls/connect/secrets/connect.cert.pem",
"content": "-----BEGIN CERTIFICATE-----\nMIIGIDCCBAigAwIBAgICEAEwDQYJKoZIhvcNAQELBQAwfTELMAkGA1UEBhMCREUx\nDzANBgNVBAgMBkJlcmxpbjEWMBQGA1UECgwNQ29uZmx1ZW50IEx0ZDELMAkGA1UE\nCwwCUFMxGDAWBgNVBAMMD0ludGVybWVkaWF0ZS1DQTEeMBwGCSqGSIb3DQEJARYP\nY2FAY29uZmx1ZW50LmlvMB4XDTE5MTExMzExMjMxOVoXDTIwMTEyMjExMjMxOVow\ngYwxCzAJBgNVBAYTAkRFMQ8wDQYDVQQIDAZCZXJsaW4xDzANBgNVBAcMBkJlcmxp\nbjEWMBQGA1UECgwNQ29uZmx1ZW50IEx0ZDELMAkGA1UECwwCUFMxFjAUBgNVBAMM\nDWthZmthLWNvbm5lY3QxHjAcBgkqhkiG9w0BCQEWD2NhQGNvbmZsdWVudC5pbzCC\nASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMjwmO0YAZV/k5DgfC48DZsp\ndjQixKEi6TLAdt2CovemFGe38DhltaIs+BiGl9ZpTC1WTbNnkwqFm3N+7+cg4I3d\nv4nYQw4RN68os9vQ+Jp/jIdVSXDP7n17pw8YOhrhQx5XMnudtghCHtp+rwG8a5KF\nIKe9zpWeu2mXlF1LxrWybbaBzJ8E7u4Gsr+suhED5dH9ckFCnLk0/9NMym7XSMnK\n/158JKhmElCanZmLLGwq+38ko6C/BgPbdaRwlKG+tHWY9Iqrt+tRHgvXclutMHci\nZgiAApS4pqOey5MWisb8yZs5SP14x8wAzyygLeNGr7+CcP+Ubcn71FvwhSodQ7EC\nAwEAAaOCAZgwggGUMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgbAMDMGCWCG\nSAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBTZXJ2ZXIgQ2VydGlmaWNhdGUw\nHQYDVR0OBBYEFAYy4GQt1eh6QaxmF4OyFBsK8xGWMIGvBgNVHSMEgacwgaSAFCbH\nLBsyO6ixqV7BnY0bLxOjVxtBoYGHpIGEMIGBMQswCQYDVQQGEwJERTEPMA0GA1UE\nCAwGQmVybGluMQ8wDQYDVQQHDAZCZXJsaW4xFjAUBgNVBAoMDUNvbmZsdWVudCBM\ndGQxCzAJBgNVBAsMAlBTMQswCQYDVQQDDAJjYTEeMBwGCSqGSIb3DQEJARYPY2FA\nY29uZmx1ZW50LmlvggIQADAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYB\nBQUHAwIGCCsGAQUFBwMBMCkGA1UdHwQiMCAwHqAcoBqGGGh0dHA6Ly9odHRwZDo4\nMC9jcmxzLnBlbTAUBgNVHREEDTALgglsb2NhbGhvc3QwDQYJKoZIhvcNAQELBQAD\nggIBAIpE7+Cu5jwv1u+TpQb2qxmR5kt6W90GySiSW0Ky+NQUL9WpbTYuGE6avpO/\n83hIf620JGTwgnwVULBaC9i8+wux7stM2vPLOaC54eDYtKgY91wWnmt061UZsie6\nFgqpNnev2U8WukxdFWdjvNQ3sWnz/aeTiaeQYmuZjSjm/sgwh92pSGqpYnuGdyc+\ne/dXxkV+lRHbMdRMoQi2ONdNQ+UZ1tJVBLjpRRaa4mq6/uemYCZ8T/BLs6Tuob2C\nNmC/x6LAntbMUaHD8PGWNlqoX//XGKwZlUvn9h+eVHgjBBbdl+69IJG6UXx7T+sV\ni3dY1RhQewfOAAdh6nFHL3tHfpkH+RnsEswq5DJXQtwQXGwtUv6fu9Aq/J+dGlua\nYpXDwdTZYCh6i2H8WfbPFAAgZfJEupNuC5EJh/BMoYEcCkLABs7ZHgQs5xVzHL/i\nli51D8gu1IqcH3+JQfQ74knDp5tWxvhQe+4m5rwct1D8IvTEOeqclvbPI4tzHmNF\nrNbk7lSCicq9axGuyrUB/Y8zDnCNThhnxjQdHm1rhJYi1IXV4BvwSBpzLocjVpRa\nT8C+fzAtk/VejGBf4zHjnu8xy7AO5hrF+ho7lAhIWZ7bVe7sZCe9D/+xgkxLP1uA\nKKhCtMklziYuNoRMpZ+be9e2OHu7p05Yj4AjhHTyMvizQLm2\n-----END CERTIFICATE-----\n"
},
{
"path": "kafka-connect-mtls/connect/secrets/connect.key.pem",
"content": "-----BEGIN RSA PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: AES-256-CBC,03578ECB28A28BA14408AF4EB2F82DCB\n\nhQkZlH8VnKBOMDtLMQT85fvYzoBYCEyjif/1/6QPid1asfxp7LGAiIEVM/sSecRz\naxGneC1B9dp1gEY09XISa2ChZwGY+vUfmpOdpyi+aeSLUeF05jsPS16fbvDfo8t4\nJnMk8XsvXs1SmXBtMvcFDnqVxWYp/kBNk2CrM24/aRgN14pykSEu0Uy5NhUGbQ/D\nvdhFq1BdWOapQQN0Oi7O5vtttOapiLKPPKaewGGV1LhyG2rJEoxj5JzrpIj2N6mB\nyjzGaHkNdvFi2BoHb/UIAaKSX5Kn+B1MW4m7eJ3YuogvBnlmr6pZaa9xxYOI+fV8\neM1JG0U1P477dqLJLRCRTtYZACmXbRy5/WV1TewKjw/ij7QHQ68ISQa2748X1gL6\n09jL5Grux/gJSDuJrhuSMyPwYSJNx5585/HLLQseKOFdvqFbHAjfd2/ZptaBpxp4\njbkylbVxvroPZQjRhj1p0v3CkUCWYUg3CSkzNLR5Y21AvqH5ZHCbjerfZvrPp3Wc\nnHShzc18wUruHmT8dwdDSb7s5OJRFkEDLNFsCsijtl47yDwaQoazeJ8UXkQ/q+FR\niIfctz0JZrWXbH89nr3i3cjwGOxQmPuMiCypYov0YezuWwyiqQMM2r62nyFTCUQQ\nooqh3OV02suBNn2GXnrXwzdCoCgcL61a2l4+rHu8rsKHKX3VzEk+SP/WsgO65KAg\njCSRV3zVTWTRbXmvFM+tv6ARDKgvhJUYAC4zuP5ZQJHsoLXhsJ/nHjlUVFVub9aT\n+BieN64UHih7lfKzK3OZJAuu5hSMY8vA8JuAkBoNNKB+CEwQnakhEQB6u87s9xgO\nGBu6med8u0isWI9uSwZ0u2/MaELRmcx6MvjdOFh8TIWU1vdtcf8F8avyP9wxGBMq\nPeFxYJ+qCx8tRUHgO6QmE/cZKmQr//mtZgOMjVwVnysmQLh8Shn1WW2FHhqWauAo\nFC2PJPw4aribkG8/O/mVx0P5/bcgHL8N7S1DWLUFsMzMYJJ35CNOG94cOFSAWFRr\n2mJLRzJFwxuh1S9S/SwqMhdz96I4OsmKAVp6iKDVusR6qBoAAjkbLCAJqF883FHo\nUrfgr5lLx/9wG6E+zGgQOC1bKsFTlSEVQ7vVLizG2etDytOvGM70Gz5ecyDY1zFl\n1D9hGLhoJtFWE8U4CLPI6pQrKXRftnUV2RbvmsgRBuHBm/HeBLbNBsuDNK3WCzvI\nYRzBpyXOblcENvj258yVtfqjRAR2b5hWeRjdyCZjxNq2S7f1Qow0nhPxe2Fq3JLu\nnGstpUt1gwKNstoMEVwYI8TzFP4kRzx5H3w2EgjzxWoybJXqANW3XHySBMcim6NR\nQMnn30bcjMI8vIe1AaL+AKskNBf4aVj+4IzvC6L+1yrzI5l4KfWbcJJk+q/rTXdQ\nmwy4DW5LfenlZoh8zQIGdHKAbdrFwI0gk0pX3Bjy69+1QAy1gNPqe5L9IUMmbsZE\nhueSRSsPgI7PDT2hv8XeoWuy+Un6/l4E34F2WvtR802kaYwgeRZIcJrFV8+yALvt\nawVcFBkjmWFRjGLFG7/f29+n998g31FqynKU9NmPL49aB8UfQBrtLY07f6snYPA2\n-----END RSA PRIVATE KEY-----\n"
},
{
"path": "kafka-connect-mtls/docker-compose.yml",
"content": "---\nversion: '3'\nservices:\n zookeeper:\n image: confluentinc/cp-zookeeper:5.3.1\n environment:\n ZOOKEEPER_CLIENT_PORT: 2181\n ZOOKEEPER_TICK_TIME: 2000\n\n kafka:\n image: confluentinc/cp-enterprise-kafka:5.3.1\n depends_on:\n - zookeeper\n ports:\n - 9092:9092\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT\n KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092\n KAFKA_AUTO_CREATE_TOPICS_ENABLE: \"true\"\n KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter\n KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1\n KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 100\n CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: kafka:29092\n CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:2181\n CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1\n CONFLUENT_METRICS_ENABLE: 'true'\n CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'\n\n schema-registry:\n image: confluentinc/cp-schema-registry:5.3.1\n depends_on:\n - zookeeper\n - kafka\n ports:\n - 8081:8081\n environment:\n SCHEMA_REGISTRY_HOST_NAME: schema-registry\n SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181\n\n kafka-connect-cp:\n image: confluentinc/cp-kafka-connect:5.3.1\n container_name: kafka-connect\n depends_on:\n - zookeeper\n - kafka\n - schema-registry\n ports:\n - 18083:18083\n volumes:\n - ./connect/secrets:/etc/kafka-connect/secrets\n environment:\n CONNECT_BOOTSTRAP_SERVERS: \"kafka:29092\"\n CONNECT_REST_PORT: 18083\n CONNECT_GROUP_ID: kafka-connect-cp\n CONNECT_CONFIG_STORAGE_TOPIC: docker-kafka-connect-cp-configs\n CONNECT_OFFSET_STORAGE_TOPIC: docker-kafka-connect-cp-offsets\n CONNECT_STATUS_STORAGE_TOPIC: docker-kafka-connect-cp-status\n CONNECT_KEY_CONVERTER: io.confluent.connect.avro.AvroConverter\n CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'\n CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter\n CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'\n CONNECT_INTERNAL_KEY_CONVERTER: \"org.apache.kafka.connect.json.JsonConverter\"\n CONNECT_INTERNAL_VALUE_CONVERTER: \"org.apache.kafka.connect.json.JsonConverter\"\n CONNECT_REST_ADVERTISED_HOST_NAME: \"kafka-connect-cp\"\n CONNECT_LOG4J_ROOT_LOGLEVEL: \"INFO\"\n CONNECT_LOG4J_LOGGERS: \"org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR\"\n CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: \"1\"\n CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: \"1\"\n CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: \"1\"\n CONNECT_PLUGIN_PATH: '/usr/share/java'\n CONNECT_LISTENERS: 'https://0.0.0.0:18083'\n CONNECT_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: \"HTTPS\"\n CONNECT_LISTENERS_HTTPS_SSL_TRUSTSTORE_LOCATION: /etc/kafka-connect/secrets/server.truststore\n CONNECT_LISTENERS_HTTPS_SSL_TRUSTSTORE_PASSWORD: confluent\n CONNECT_LISTENERS_HTTPS_SSL_KEYSTORE_LOCATION: /etc/kafka-connect/secrets/server.keystore\n CONNECT_LISTENERS_HTTPS_SSL_KEYSTORE_PASSWORD: confluent\n CONNECT_LISTENERS_HTTPS_SSL_CLIENT_AUTH: required\n"
},
{
"path": "kafka-connect-mtls/up",
"content": "#!/usr/bin/env bash\n\ndocker-compose up -d\n\necho \"to verify the connection use the check-ssl-client-auth.sh script\"\n"
},
{
"path": "kerberos/README.md",
"content": "# Kerberos configuration demo\n\nThis demo sets up a basic Kafka cluster secured with Kerberos authentication, and sets up some basic ACLs to demonstrate authorisation.\nThe documentation below introduces the relevant components you need to understand to set up Kerberos in a Linux / JVM environment.\n\n\n## Kerberos authentication process\n\nBefore configuring Kafka for Kerberos authentication, it is useful to understand the basics of Kerberos; the authentication process and some key terms.\n\n_A note on what Kerberos is and is not: Kerberos is a *network authentication protocol* which allows a client application to connect to a network service in a way that allows the components to mutually verify each other's identities._\n_It is put to good use in and integrated with network directory services, notably Windows Active Directory._\n_Here, Kerberos identities are bound to network accounts and access privileges and, in the case of Windows, the SSPI API supports single sign-on and privilege impersonation natively in the OS._\n_This is enabled by Kerberos, but Kerberos itself is not bound to such accounts and does not provide any such capability._\n\nWith that, let's work through the process for a client application making a connection into Kafka.\n\nKerberos involves three parties:\n\n- a Kerberos Client, in this case our client application.\n- a Kerberized Service, in this case Kafka.\n- the Kerberos **Key Distribution Center (KDC)**\n\nAn important point to understand in this process is that the Client and Service each shares their own cryptographic key with the KDC.\nBy using this key to encrypt/decrypt tokens passed over the network, two network systems can verify each other's identities.\nThe Client and Service trust that they have only shared their secret with the KDC and so any correctly signed token must have originated from the KDC.\nThis is crucial.\nDuring the Kerberos process the Client requests a token from the KDC _signed with the Service's key_ and presents this when making a connection.\nThe Service can then trust that the Client has valid credentials with the KDC and can be authenticated.\n\nOther information is shared during the process to enable integrity checking and protection against various spoofing attacks.\nFor example, each signed token is:\n\n* timestamped to bound the window for which it is valid\n* linked to a network IP so that it is valid only from a single host\n\nThe first stage is that the Client application must authenticate itself with the KDC by proving that the Client knows the private credentials relating to the Client's Kerberos **Principal**.\nThe Principal is a a unique identity in the form {primary}/{instance}@{REALM} (more on these later).\nThe KDC authenticates the client using their shared cryptographic key and results in the client receiving a **Ticket Granting Ticket (TGT)**.\nThis is a cryptographic token that the Client may now use to prove that it has recently authenticated with the KDC.\nThe TGT is timestamped and includes an expiry time, typically a day.\nThe TGT is cached by the client to avoid having to re-authenticate unnecessarily.\n\nNext, the Client wants to authenticate itself to the Kerberized Service.\nFor this to happen, the client must get a cryptographic token encrypted with the Kerberized Service's key - this token is a **Service Ticket** and is requested by the client from the KDC using the TGT and the requested service's principal name.\nIncluding the TGT in this request is sufficient to prove that the client has already authenticated with the KDC allowing the service ticket to be returned.\n\nHere is an important point to note - how does the client know the service principal name?\nVery simply, it builds the principal with:\n\n- {primary} = a client-side configured name for the service\n- {instance} = the network address used to connect to the service\n- {REALM} = the realm of the client and KDC.\n\nIn our example, our Client attempts to connect to the `kafka` Service on the host `kafka.kerberos-demo.local` in the realm `TEST.CONFLUENT.IO`.\nTherefore, the service must be configured with a Service Principal Name of `kafka/kafka.kerberos-demo.local@TEST.CONFLUENT.IO`.\n\nNow the Client can connect directly to the Kerberized Service, and include the Service Ticket.\nAs the Service ticket is signed with the Service principal's key, the Service can decrypt the token to authenticate the request.\n\nBased on the above, each connection in the cluster must be established with the following in place:\n\n* On the Kerberos Client:\n * A client principal and key to authenticate with the KDC, `{client name}@REALM`\n * This is the *User Principal Name*.\n * a configured name for the service to connect to, `{service name}`\n * the network address for the service, `{network address}`. \n* On the server:\n * a principal name & key in the form `{service name}/{network address}@REALM`.\n * This is the *Service Principal Name*.\n\nAs can be seen, the service principal must be constructed correctly to work.\nHowever, the `{client name}` format is not mandated in the same way and is not bound to a network address.\nOften the client name is a simple alphanumeric username, let's say 'john'.\nHowever, you may sometimes see a client principal such as 'john/admin'.\nIn this form, 'admin' is called an _instance_ of the 'john' principal and can be used by 'john' to run services on the system with different credentials and privileges from the main account.\nFrom the Kerberos perspective, the two principals are completely separate, but it can nonetheless be convenient to use this naming convention.\n\n\n# Technical Components\n## KDC\nThe KDC could be provided by MIT Kerberos, Windows Active Directory, Redhat Identity Manager and many others.\nIn this demo we use MIT kerberos.\n\n\n## Kerberos libraries and tools\n\nAll the hosts must include Kerberos libraries and a shared configuration (krb5.conf) in order to use and trust the same KDC.\n\n`kinit` is used to authenticate to the Kerberos server as principal, or if none is given, a system generated default (typically your login name at the default realm), and acquire a ticket granting ticket that can later be used to obtain tickets for other services.\n\n`klist` reads and displays the current tickets in the credential cache (also known as the ticket file).\n\n`kvno` acquires a service ticket for the specified Kerberos principals and prints out the key version numbers of each.\n\n`kadmin` is an admin utility for working with the Kerberos database.\n\nA common task when configuring for Kerberos is to build *keytab* files (short for Key Table).\nKeytabs are files containing one or more Kerberos principal/credential pairs.\nBy having these in a file, services can automatically authenticate with the KDC without prompting the user and it is common to build and distribute keytabs as part of a deployment.\nHowever, _as these files contain secret credentials, it is important to take care to protect against loss of these files_.\n\nSee [kerberos cheatsheet](../KerberosCheatsheet.md) for examples of using the Kerberos toolset.\n\n\n## Simple Authentication and Security Layer (SASL)\n\nSASL is a framework for authentication in network communications which in principle decouples authentication concerns from the application protocol.\n\nKafka and Zookeeper can use SASL as the authentication layer in communications (Mutual TLS being the notable alternative).\n\nWhen SASL has been enabled, you must further specify a SASL *mechanism* to use - the process and protocol to use when authenticating a connection.\nApplications must build support for each SASL mechanism - Kafka supports SCRAM(-SHA-256 | -SHA-512), PLAIN, OAUTHBEARER and GSSAPI.\n*GSSAPI is the SASL mechanism which implements Kerberos*.\n\n\n## Java Authentication and Authorization Services (JAAS)\n\nJAAS is a Java's integrated, pluggable security service and Kafka uses the JAAS APIs to implement SASL authentication.\nSASL authentication is configured using JAAS.\nKerberos is configured using the JAAS *LoginModule* `com.sun.security.auth.module.Krb5LoginModule`.\n\nJAAS may be configured in a couple of places:\n\n* By default it uses a .jaas file, a reference to which is passed in the `-Djava.security.auth.login.config=` JVM flag.\nEach jaas file includes multiple named stanzas, representing different login contexts.\n\n* An application can override this configuration and configure JAAS from application config.\nKafka configurations expose this option using properties `sasl.jaas.config`, which can variously be prefixed.\nThe value is the inline configuration for a single login context and, in Kafka, takes precedence over entries in a .jaas files.\n\nhttps://docs.oracle.com/javase/8/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html\n\n\nA Kerberos enabled Client or Service can be initiated in two ways:\n\n1. Use `kinit` to cache a TGT locally, and then launch the process with this shared cache.\n2. Configure a keytab to be used directly.\n\nConfiguration of the former is straight-forward as follows:\n\n```\nSomeLoginContext {\n com.sun.security.auth.module.Krb5LoginModule required\n useTicketCache = true;\n};\n```\n\nThe `useTicketCache = true` setting specifies that the TGT cache should be used.\n\nBy comparison, the latter approach has `useTicketCache = false` (the default) and then continues to specify details for using a keytab file:\n\n```\nSomeLoginContext {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/var/lib/secret/kafka1.key\"\n principal=\"kafka/kafka1.kerberos-demo.local@TEST.CONFLUENT.IO\";\n};\n```\n\nThe login context, as identified with `SomeLoginContext` above, can be used by a Client, a Service or both.\nFor Kafka, the names are defined in the application code as we will describe later.\n\n\n# Kerberizing Kafka\n\nTo fully understand the steps required to Kerberize Kafka, we should understand each Client → Service connection which we wish to configure.\nEach of these connections has a prototypical set of configurations required on the Client side and on the Service side.\n\nThe following are values you must decide upon at the cluster level:\n\n* `{kafka-kerberos-service-name}` - name for the Kerberized Kafka service.\nTypically `kafka` or `cp-kafka`.\n* `{zookeeper-kerberos-service-name}` - name for the Kerberized Zookeeper service.\nBy default this is `zookeeper`.\n* `{security-protocol}` - either `SASL_PLAINTEXT` of `SASL_SSL` if using in conjunction with TLS.\n\n## Service Configurations\n\nIn a single node Broker/Zookeeper environment there are just two Kerberized services running.\nWe will configure these first and then the clients.\n\n### Kafka Service\n\n* Broker JAAS:\n * Login Context: `KafkaServer`\n * Use *keytab* method.\n * Ensure that the principal is a correctly formed service principal for each node: `{kafka-kerberos-service-name}/{FQDN}@{realm}`.\n * Example: [kafka/kafka.sasl.jaas.config](kafka/kafka.sasl.jaas.config)\n* Broker Server Properties:\n * `sasl.enabled.mechanisms=GSSAPI` (more SASL mechanisms may be specified in a comma-separated list)\n * `sasl.kerberos.service.name={kafka-kerberos-service-name}`\n * `{listener_name}.{sasl_mechanism}.sasl.jaas.config` - jaas configuration on a per-listener basis.\n * Example: [kafka/server.properties](kafka/server.properties)\n\n### Zookeeper Service\n\n* Zookeeper JAAS:\n * Client API - Kerberize access to ZooKeeper data.\n * Login Context: `Server`.\n * Use *keytab* method.\n * Ensure that the principal is a correctly formed service principal for each node: `{zookeeper-kerberos-service-name}/{FQDN}@{realm}`.\n * Example: [zookeeper/zookeeper.sasl.jaas.config](zookeeper/zookeeper.sasl.jaas.config)\n* Zookeeper Properties:\n * authProvider.1 = org.apache.zookeeper.server.auth.SASLAuthenticationProvider\n * requireClientAuthScheme=sasl\n * Example: [zookeeper/zookeeper.properties](zookeeper/zookeeper.properties)\n\n\n## Client Configurations\n\n\n### Kafka Broker → Zookeeper Service\nBrokers connect to Zookeeper for cluster operations.\n\n* Broker JAAS:\n * Login Context: `Client`\n * Use *keytab* method.\n * *Ensure that the same principal is configured for use on each broker.*\n * Example: [kafka/kafka.sasl.jaas.config](kafka/kafka.sasl.jaas.config).\n* Broker JVM flags:\n * `-Dzookeeper.sasl.client.username={zookeeper-kerberos-service-name}` (OPTIONAL)\n\n\n### Client → Kafka Service\nClients connecting in to Kafka may be any of:\n\n* A Kafka producer\n* A Kafka consumer\n* A Kafka Admin client\n\nNote that many applications are a combination of many of these - notably Streams applications and Kafka Connect.\n\n* Client JAAS:\n * Login Context: `KafkaClient`\n * Can use *kinit* or *keytab* method.\n * Example: [client/client.sasl.jaas.config](client/client.sasl.jaas.config)\n* Client Properties:\n * `sasl.kerberos.service.name={kafka-kerberos-service-name}`\n * `security.protocol={security-protocol}`\n * `sasl.jaas.config` - jaas override.\n * Examples: [client/producer.properties](client/producer.properties), [client/consumer.properties](client/consumer.properties), [client/command.properties](client/command.properties)\n\n\n### Client → Zookeeper (Optional)\nHistorically, clients needed to connect directly to ZooKeeper for service discovery and admin operations.\nHowever, the new Kafka Admin API allows all this functionality via Client → Kafka Broker connection, so this direct connection should not be required.\n\n* JAAS:\n * LoginContext: `Client`\n * Can use *kinit* or *keytab* method.\n\n * JVM flags:\n * `-Dzookeeper.sasl.client.username={zookeeper-kerberos-service-name}` (OPTIONAL)\n\n### Confluent Metrics Reporter → Kafka Service (Optional)\n\nThe Confluent metrics reporter runs as a plugin within the Kafka broker, but from a Kerberos point of view is configured as a network client.\nThe configuration, including inline Jaas, is specified within the broker properties using a keytab:\n\n* `confluent.metrics.reporter.sasl.mechanism=GSSAPI`\n* `confluent.metrics.reporter.security.protocol={security-protocol}`\n* `confluent.metrics.reporter.sasl.kerberos.service.name={kafka-kerberos-service-name}`\n* `confluent.metrics.reporter.sasl.jaas.config={inline jaas configuration}`\n* Example: [kafka/server.properties](kafka/server.properties)\n\n# Authentication is not enough!\n\nThe steps above are sufficient to support Kerberos authenticated connections within the cluster.\nThis does not make your cluster secure though!\nThe demo also applies a minimal level of authorisation to prevent unauthenticated network access to the brokers and Zookeeper.\nThe following should be reviewed in the broker server properties:\n\n* `allow.everyone.if.no.acl.found=false` - default to no access.\n* `authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer` - enable the default authoriser for Kafka.\n* `zookeeper.set.acl=true` - when storing ACL data in Zookeeper, apply Zookeeper access controls so that only the Broker → Zookeeper client principal can read and modify the lists.\n* Example: [kafka/server.properties](kafka/server.properties)\n\n# Putting it into action\n\nIn this demo we configure:\n\n* A simple KDC to generate principals and keytabs.\n* A single node Zookeeper with a Kerberized data access API.\n* A single node Kafka broker with a Kerberized listener.\n* Set up ACLs allowing `kafka-console-producer` and `kafka-console-consumer` usage.\n\n_A basic knowledge of Docker is useful to follow the code, though only basic Docker techniques are used to keep the code readable._\n_Each node is built using a `Dockerfile` into which configuration values are hard-coded, and the services are brought up using `docker-compose`._\n_Kerberos keytabs and the krb5.conf file are shared amongst all nodes on the cluster using a shared Docker volume._\n\nThe demo is run using the [up](up) script, which orchestrates the following process:\n\n* Builds and starts the KDC.\nAll nodes are joined to the KDC's realm by sharing `krb5.conf` amongst all nodes.\n* Generates Kerberos principals and keytabs, sharing these on the shared Docker volume.\n* Builds and starts Zookeeper, Kafka broker and Client.\n* Uses the `admin` super user to configure ACLs for the `producer` and `consumer` users.\n* Prints example usage to connect into Kafka with a Kerberos principal.\nThis is actually executed via the `client` node.\n\n# Next up\n\n* Extending Kerberos configuration to a full cluster (coming soon)\n* Hardening access controls\n\n\n# References\n\n* https://www.youtube.com/watch?v=KD2Q-2ToloE Video overview of Kerberos authentication process.\n"
},
{
"path": "kerberos/client/Dockerfile",
"content": "FROM centos:centos7\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.4/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install confluent kafka tools:\nRUN yum install -y java-11-openjdk\nRUN yum install -y confluent-kafka-2.12\n\n# 3. Install Kerberos libaries\nRUN yum install -y krb5-workstation krb5-libs\n\n# 4. Copy in required settings for client access to Kafka\nCOPY consumer.properties /etc/kafka/consumer.properties\nCOPY producer.properties /etc/kafka/producer.properties\nCOPY command.properties /etc/kafka/command.properties\nCOPY client.sasl.jaas.config /etc/kafka/client_jaas.conf\n\nENV KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf\n\nCMD sleep infinity\n"
},
{
"path": "kerberos/client/client.sasl.jaas.config",
"content": "/*\n* Credentials to use when connecting to ZooKeeper directly.\n*\n* Whenever possible you should use the Kafka AdminClient API instead of ZooKeeper.\n*/\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useTicketCache=true;\n};\n\n\n/*\n* Credentials to connect to Kafka.\n*/\nKafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required \n useTicketCache=true;\n};\n"
},
{
"path": "kerberos/client/command.properties",
"content": "bootstrap.servers=kafka:9093\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \\\n\t\t\t\t\t\t\t\t serviceName=kafka \\\n\t\t\t\t\t\t\t\t useTicketCache=true;\n"
},
{
"path": "kerberos/client/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.4/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.4\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n"
},
{
"path": "kerberos/client/consumer.properties",
"content": "bootstrap.servers=kafka:9093\nsecurity.protocol=SASL_PLAINTEXT\nsasl.kerberos.service.name=kafka\nsasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \\\n\t\t\t\t\t\t\t\t useTicketCache=true;\n"
},
{
"path": "kerberos/client/producer.properties",
"content": "bootstrap.servers=kafka:9093\nsecurity.protocol=SASL_PLAINTEXT\nsasl.kerberos.service.name=kafka\nsasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \\\n\t\t\t\t\t\t\t\t useTicketCache=true;\n"
},
{
"path": "kerberos/docker-compose.yml",
"content": "version: '3.5'\nservices:\n kdc:\n hostname: kdc.kerberos-demo.local\n #domainname: kerberos_default\n build: kdc/\n container_name: kdc\n volumes:\n - secret:/var/lib/secret\n - ./kdc/krb5.conf:/etc/kdc/krb5.conf\n\n zookeeper:\n build: zookeeper/\n container_name: zookeeper\n hostname: zookeeper.kerberos-demo.local\n #domainname: kerberos_default\n depends_on:\n - kdc\n # Required to wait for the keytab to get generated\n restart: on-failure\n volumes:\n - secret:/var/lib/secret\n - ./kdc/krb5.conf:/etc/krb5.conf\n\n kafka:\n build: kafka/\n container_name: kafka\n hostname: kafka.kerberos-demo.local\n #domainname: kerberos_default\n depends_on:\n - zookeeper\n - kdc\n # Required to wait for the keytab to get generated\n restart: on-failure\n volumes:\n - secret:/var/lib/secret\n - ./kdc/krb5.conf:/etc/krb5.conf\n\n client:\n build: client/\n container_name: client\n hostname: client.kerberos-demo.local\n #domainname: kerberos_default\n depends_on:\n - kafka\n - kdc\n # Required to wait for the keytab to get generated\n volumes:\n - secret:/var/lib/secret\n - ./kdc/krb5.conf:/etc/krb5.conf\n\nvolumes:\n secret: {}\n\nnetworks:\n default:\n name: kerberos-demo.local\n"
},
{
"path": "kerberos/kafka/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.4/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-11-openjdk\nRUN yum install -y confluent-kafka-2.12\nRUN yum install -y confluent-control-center\n\n# 3. Configure Kafka for Kerberos\nRUN yum install -y krb5-workstation krb5-libs\nCOPY server.properties /etc/kafka/server.properties\nCOPY kafka.sasl.jaas.config /etc/kafka/kafka_server_jaas.conf\n\nEXPOSE 9093\n\nENV KAFKA_OPTS=\"-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf -Dzookeeper.sasl.client.username=zkservice\"\n\nCMD kafka-server-start /etc/kafka/server.properties\n"
},
{
"path": "kerberos/kafka/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.4/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.4\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n"
},
{
"path": "kerberos/kafka/kafka.sasl.jaas.config",
"content": "/*\n* The service principal\n*/\nKafkaServer {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/var/lib/secret/kafka.key\"\n principal=\"kafka/kafka.kerberos-demo.local@TEST.CONFLUENT.IO\";\n};\n\n/*\n* Zookeeper client principal\n*/\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n\t\tuseTicketCache=false\n keyTab=\"/var/lib/secret/zookeeper-client.key\"\n principal=\"zkclient@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kerberos/kafka/server.properties",
"content": "# Basic broker and listener configuration\nbroker.id=0\nlisteners=SASL_PLAINTEXT://kafka.kerberos-demo.local:9093\nzookeeper.connect=zookeeper.kerberos-demo.local:2181\nlog.dirs=/var/lib/kafka\n\noffsets.topic.replication.factor=1\ntransaction.state.log.replication.factor=1\ntransaction.state.log.min.isr=1\nnum.partitions=12\n\n\n# Kerberos / GSSAPI Authentication mechanism\nsasl.enabled.mechanisms=GSSAPI\nsasl.kerberos.service.name=kafka\n\n\n# Configure replication to require Kerberos:\nsasl.mechanism.inter.broker.protocol=GSSAPI\nsecurity.inter.broker.protocol=SASL_PLAINTEXT\n\n\n# Authorization config:\nauthorizer.class.name=kafka.security.auth.SimpleAclAuthorizer\nzookeeper.set.acl=true\nallow.everyone.if.no.acl.found=false\nsuper.users=User:admin;User:kafka\n\n\n# Demonstrate setting up the Confluent Metrics Reporter with required *client* credentials\nmetric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter\nconfluent.metrics.reporter.bootstrap.servers=kafka:9093\nconfluent.metrics.reporter.sasl.mechanism=GSSAPI\nconfluent.metrics.reporter.security.protocol=SASL_PLAINTEXT\nconfluent.metrics.reporter.sasl.kerberos.service.name=kafka\nconfluent.metrics.reporter.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \\\n useKeyTab=true \\\n storeKey=true \\\n keyTab=\"/var/lib/secret/kafka-admin.key\" \\\n principal=\"admin/for-kafka@TEST.CONFLUENT.IO\";\n\nconfluent.metrics.reporter.topic.replicas=1\nconfluent.support.metrics.enable=false\nconfluent.support.customer.id=anonymous\n"
},
{
"path": "kerberos/kdc/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Installing Kerberos server, admin and client\nRUN yum install -y krb5-server krb5-libs \nRUN yum install -y krb5-workstation krb5-libs \n\n# 2. Configuring Kerberos and KDC\nCOPY krb5.conf /etc/krb5.conf\nRUN mkdir /var/log/kerberos\nRUN mkdir /etc/kdc\nRUN mkdir -p /var/kerberos/krb5kdc/ \nRUN ln -s /etc/krb5.conf /etc/kdc/krb5.conf\n\nEXPOSE 88\n\nRUN kdb5_util -P confluent -r TEST.CONFLUENT.IO create -s\n\nCMD /usr/sbin/krb5kdc -n\n"
},
{
"path": "kerberos/kdc/krb5.conf",
"content": "[libdefaults]\n default_realm = TEST.CONFLUENT.IO\n forwardable = true\n rdns = false\n dns_lookup_kdc = no\n dns_lookup_realm = no\n\n[realms]\n\tTEST.CONFLUENT.IO = {\n\t\tkdc = kdc\n\t\tadmin_server = kadmin\n\t}\n\n[domain_realm]\n\t.test.confluent.io = TEST.CONFLUENT.IO\n\ttest.confluent.io = TEST.CONFLUENT.IO\n kerberos-demo.local = TEST.CONFLUENT.IO\n .kerberos-demo.local = TEST.CONFLUENT.IO\n\n[logging]\n\tkdc = FILE:/var/log/kerberos/krb5kdc.log\n\tadmin_server = FILE:/var/log/kerberos/kadmin.log\n default = FILE:/var/log/kerberos/krb5lib.log\n"
},
{
"path": "kerberos/up",
"content": "#!/bin/sh\nset -e\n\n# Starting kerberos,\n# Avoiding starting up all services at the begining to generate the keytab first\n\ndocker-compose build\ndocker-compose up -d kdc\n\n### Create the required identities:\n# Kafka service principal:\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey kafka/kafka.kerberos-demo.local@TEST.CONFLUENT.IO\" > /dev/null\n\n# Zookeeper service principal:\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey zkservice/zookeeper.kerberos-demo.local@TEST.CONFLUENT.IO\" > /dev/null\n\n# Create a principal with which to connect to Zookeeper from brokers - NB use the same credential on all brokers!\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey zkclient@TEST.CONFLUENT.IO\" > /dev/null\n\n# Create client principals to connect in to the cluster:\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey kafka_producer@TEST.CONFLUENT.IO\" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey kafka_producer/instance_demo@TEST.CONFLUENT.IO\" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey kafka_consumer@TEST.CONFLUENT.IO\" > /dev/null\n\n# Create an admin principal for the cluster, which we'll use to setup ACLs.\n# Look after this - its also declared a super user in broker config.\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey admin/for-kafka@TEST.CONFLUENT.IO\" > /dev/null\n\n# Create keytabs to use for Kafka\ndocker exec -ti kdc rm -f /var/lib/secret/kafka.key 2>&1 > /dev/null\ndocker exec -ti kdc rm -f /var/lib/secret/zookeeper.key 2>&1 > /dev/null\ndocker exec -ti kdc rm -f /var/lib/secret/zookeeper-client.key 2>&1 > /dev/null\ndocker exec -ti kdc rm -f /var/lib/secret/kafka-client.key 2>&1 > /dev/null\ndocker exec -ti kdc rm -f /var/lib/secret/kafka-admin.key 2>&1 > /dev/null\n\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/kafka.key -norandkey kafka/kafka.kerberos-demo.local@TEST.CONFLUENT.IO \" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/zookeeper.key -norandkey zkservice/zookeeper.kerberos-demo.local@TEST.CONFLUENT.IO \" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/zookeeper-client.key -norandkey zkclient@TEST.CONFLUENT.IO \" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/kafka-client.key -norandkey kafka_producer@TEST.CONFLUENT.IO \" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/kafka-client.key -norandkey kafka_producer/instance_demo@TEST.CONFLUENT.IO \" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/kafka-client.key -norandkey kafka_consumer@TEST.CONFLUENT.IO \" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/kafka-admin.key -norandkey admin/for-kafka@TEST.CONFLUENT.IO \" > /dev/null\n\n\n# Starting zookeeper and kafka now that the keytab has been created with the required credentials and services\ndocker-compose up -d\n\n# Adding ACLs for consumer and producer user:\ndocker exec client bash -c \"kinit -k -t /var/lib/secret/kafka-admin.key admin/for-kafka && kafka-acls --bootstrap-server kafka:9093 --command-config /etc/kafka/command.properties --add --allow-principal User:kafka_producer --producer --topic=*\"\ndocker exec client bash -c \"kinit -k -t /var/lib/secret/kafka-admin.key admin/for-kafka && kafka-acls --bootstrap-server kafka:9093 --command-config /etc/kafka/command.properties --add --allow-principal User:kafka_consumer --consumer --topic=* --group=*\"\n\n\n# Output example usage:\necho \"Example configuration to access kafka:\"\necho \"-> docker-compose exec client bash -c 'kinit -k -t /var/lib/secret/kafka-client.key kafka_producer && kafka-console-producer --broker-list kafka:9093 --topic test --producer.config /etc/kafka/producer.properties'\"\necho \"-> docker-compose exec client bash -c 'kinit -k -t /var/lib/secret/kafka-client.key kafka_consumer && kafka-console-consumer --bootstrap-server kafka:9093 --topic test --consumer.config /etc/kafka/consumer.properties --from-beginning'\"\n"
},
{
"path": "kerberos/zookeeper/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.4/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-11-openjdk\nRUN yum install -y confluent-kafka-2.12\n\n# 3. Configure zookeeper for Kerberos\nRUN yum install -y krb5-workstation krb5-libs\nCOPY zookeeper.properties /etc/kafka/zookeeper.properties\nCOPY zookeeper.sasl.jaas.config /etc/kafka/zookeeper_server_jaas.conf\n\nEXPOSE 2181\n\nENV KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/zookeeper_server_jaas.conf\n\nCMD zookeeper-server-start /etc/kafka/zookeeper.properties\n"
},
{
"path": "kerberos/zookeeper/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.4/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.4\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n"
},
{
"path": "kerberos/zookeeper/zookeeper.properties",
"content": "dataDir=/var/lib/zookeeper\nclientPort=2181\nmaxClientCnxns=0\nauthProvider.1 = org.apache.zookeeper.server.auth.SASLAuthenticationProvider\nzookeeper.allowSaslFailedClients=false\nrequireClientAuthScheme=sasl\n"
},
{
"path": "kerberos/zookeeper/zookeeper.sasl.jaas.config",
"content": "Server {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n keyTab=\"/var/lib/secret/zookeeper.key\"\n storeKey=true\n useTicketCache=false\n principal=\"zkservice/zookeeper.kerberos-demo.local@TEST.CONFLUENT.IO\";\n};\n\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useTicketCache=true;\n};\n"
},
{
"path": "kerberos-multi-node/README.md",
"content": "# Kerberos multi-node deployment example\n\nThis example shows how-to deploy multiple kafka nodes in an example kerberos enabled environment.\n\nThe only thing that's different then your normal environment is that this example uses a different principal for each zookeeper client.\n\nhttps://issues.apache.org/jira/browse/KAFKA-7710 Jira contains a more information. \nTLDR; we have to set two configs in the zookeeper.properties to make this work\n\n```\nkerberos.removeHostFromPrincipal = true\nkerberos.removeRealmFromPrincipal = false\n```\n\nThe first removes the hostname from the principal name. \nSo that anyone authenticated with the principal 'kafka/*@REALM' is allowed by ZK ACLs.\n"
},
{
"path": "kerberos-multi-node/docker-compose.yml",
"content": "version: '3.8'\nservices:\n kdc:\n hostname: kdc\n domainname: kerberos-multi-node_default\n build: kdc/\n container_name: kdc\n volumes:\n - secret:/var/lib/secret\n\n zookeeper:\n build: zookeeper/\n container_name: zookeeper\n hostname: zookeeper\n domainname: kerberos-multi-node_default\n depends_on: \n - kdc\n # Required to wait for the keytab to get generated\n restart: on-failure\n environment:\n - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/zookeeper_server_jaas.conf -Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/tmp/krb5.conf\n - KRB5_CONFIG=/tmp/krb5.conf\n #- KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/zookeeper_server_jaas.conf\n volumes:\n - secret:/var/lib/secret\n - ./kdc/krb5.conf:/tmp/krb5.conf\n\n kafka:\n build: kafka/\n container_name: kafka\n hostname: kafka\n domainname: kerberos-multi-node_default\n environment:\n # - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf -Dsun.security.krb5.debug=true -Dzookeeper.sasl.client.username=zkservice\n - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf -Dzookeeper.sasl.client.username=zkservice -Djava.security.krb5.conf=/tmp/krb5.conf\n - KRB5_CONFIG=/tmp/krb5.conf\n depends_on: \n - zookeeper\n - kdc\n # Required to wait for the keytab to get generated\n restart: on-failure\n volumes:\n - secret:/var/lib/secret\n - ./kdc/krb5.conf:/tmp/krb5.conf\n\n kafka1:\n build: kafka1/\n container_name: kafka1\n hostname: kafka1\n domainname: kerberos-multi-node_default\n environment:\n # - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf -Dsun.security.krb5.debug=true -Dzookeeper.sasl.client.username=zkservice\n - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf -Dzookeeper.sasl.client.username=zkservice -Djava.security.krb5.conf=/tmp/krb5.conf\n - KRB5_CONFIG=/tmp/krb5.conf\n depends_on: \n - zookeeper\n - kdc\n # Required to wait for the keytab to get generated\n restart: on-failure\n volumes:\n - secret:/var/lib/secret\n - ./kdc/krb5.conf:/tmp/krb5.conf\n\nvolumes:\n secret: {}\n"
},
{
"path": "kerberos-multi-node/down",
"content": "#!/bin/bash\nDESTROY=no\nif [ ! -f \"${PWD}/docker-compose.yml\" ]; then\n echo \"No docker-compose found. Exiting.\"\n exit 2\nfi\n\nusage() \n{\n echo \"Usage: $0 [-h] [-d]\"\n echo \"-d destroy images. They will be rebuilt next time\"\n exit 2\n}\n\ndestroy() \n{\n docker-compose rm --force\n}\n\nstop_docker-compose()\n{\n docker-compose stop\n}\n\n# Should use getopts here but, why?\nif [[ \"${1}\" == \"-h\" ]]; then\n usage\n exit 2\nfi\n\nif [[ \"${1}\" == \"-d\" ]]; then\n echo \"Stopping and destroying containers\"\n DESTROY=yes\nfi\n\nstop_docker-compose \nif [[ $? != 0 ]]; then\n echo \"Stopping the docker-compose failed. Exiting for manual cleanup\"\n echo \"I suggest 'docker-compose ps'\"\n exit 2\nfi\n\nif [[ \"${DESTROY}\" == \"yes\" ]]; then\n destroy\nfi\n\n"
},
{
"path": "kerberos-multi-node/kafka/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/6.0/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-11-openjdk\nRUN yum install -y confluent-platform-2.12\nRUN yum install -y confluent-control-center\n\n# 3. Configure Kafka for Kerberos \nRUN yum install -y krb5-workstation krb5-libs \nCOPY server.properties /etc/kafka/server.properties\nCOPY kafka.sasl.jaas.config /etc/kafka/kafka_server_jaas.conf\nCOPY consumer.properties /etc/kafka/consumer.properties\n\nEXPOSE 9093\n\nCMD kafka-server-start /etc/kafka/server.properties\n"
},
{
"path": "kerberos-multi-node/kafka/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/6.0/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/6.0/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/6.0\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/6.0/archive.key\nenabled=1\n"
},
{
"path": "kerberos-multi-node/kafka/consumer.properties",
"content": "bootstrap.servers=kafka:9093\nsecurity.protocol=SASL_PLAINTEXT\nsasl.kerberos.service.name=kafka\nsasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \\\n\t\t\t\t\t\t\t\t useTicketCache=true;\n"
},
{
"path": "kerberos-multi-node/kafka/kafka.sasl.jaas.config",
"content": "KafkaServer {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/var/lib/secret/kafka.key\"\n principal=\"kafka/kafka.kerberos-multi-node_default@TEST.CONFLUENT.IO\";\n};\n\nKafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/var/lib/secret/kafka.key\"\n principal=\"admin@TEST.CONFLUENT.IO\";\n};\n\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n useTicketCache=false\n keyTab=\"/var/lib/secret/kafka.key\"\n principal=\"kafka@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kerberos-multi-node/kafka/server.properties",
"content": "broker.id=0\nlisteners=SASL_PLAINTEXT://kafka:9093\nadvertised.listeners=SASL_PLAINTEXT://kafka:9093\nsecurity.inter.broker.protocol=SASL_PLAINTEXT\nlog.dirs=/var/lib/kafka\noffsets.topic.replication.factor=1\ntransaction.state.log.replication.factor=1\ntransaction.state.log.min.isr=1\nzookeeper.connect=zookeeper.kerberos-multi-node_default:2181\nzookeeper.set.acl=true\n\n# Kerberos / GSSAPI Authentication mechanism\nsasl.enabled.mechanisms=GSSAPI\nsasl.mechanism.inter.broker.protocol=GSSAPI\nsecurity.inter.broker.protocol=SASL_PLAINTEXT\nsasl.kerberos.service.name=kafka\nallow.everyone.if.no.acl.found=false\nsuper.users=User:admin;User:kafka\nauthorizer.class.name=kafka.security.auth.SimpleAclAuthorizer\n\n# metric reporter configuration with Kerberos\nmetric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter\nconfluent.metrics.reporter.bootstrap.servers=kafka:9093\nconfluent.metrics.reporter.sasl.mechanism=GSSAPI\nconfluent.metrics.reporter.security.protocol=SASL_PLAINTEXT\nconfluent.metrics.reporter.sasl.kerberos.service.name=kafka\nconfluent.metrics.reporter.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \\\n useKeyTab=true \\\n storeKey=true \\\n keyTab=\"/var/lib/secret/kafka.key\" \\\n principal=\"kafka@TEST.CONFLUENT.IO\";\nconfluent.metrics.reporter.topic.replicas=1\n\n\n"
},
{
"path": "kerberos-multi-node/kafka1/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/6.0/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-11-openjdk\nRUN yum install -y confluent-platform-2.12\nRUN yum install -y confluent-control-center\n\n# 3. Configure Kafka for Kerberos \nRUN yum install -y krb5-workstation krb5-libs \nCOPY server.properties /etc/kafka/server.properties\nCOPY kafka.sasl.jaas.config /etc/kafka/kafka_server_jaas.conf\nCOPY consumer.properties /etc/kafka/consumer.properties\n\nEXPOSE 9093\n\nCMD kafka-server-start /etc/kafka/server.properties\n"
},
{
"path": "kerberos-multi-node/kafka1/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/6.0/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/6.0/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/6.0\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/6.0/archive.key\nenabled=1\n"
},
{
"path": "kerberos-multi-node/kafka1/consumer.properties",
"content": "bootstrap.servers=kafka:9093\nsecurity.protocol=SASL_PLAINTEXT\nsasl.kerberos.service.name=kafka\nsasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \\\n\t\t\t\t\t\t\t\t useTicketCache=true;\n"
},
{
"path": "kerberos-multi-node/kafka1/kafka.sasl.jaas.config",
"content": "KafkaServer {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/var/lib/secret/kafka.key\"\n principal=\"kafka/kafka1.kerberos-multi-node_default@TEST.CONFLUENT.IO\";\n};\n\nKafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/var/lib/secret/kafka.key\"\n principal=\"admin@TEST.CONFLUENT.IO\";\n};\n\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n useTicketCache=false\n keyTab=\"/var/lib/secret/kafka.key\"\n principal=\"kafka@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kerberos-multi-node/kafka1/server.properties",
"content": "broker.id=1\nlisteners=SASL_PLAINTEXT://kafka1:9093\nadvertised.listeners=SASL_PLAINTEXT://kafka1:9093\nsecurity.inter.broker.protocol=SASL_PLAINTEXT\nlog.dirs=/var/lib/kafka\noffsets.topic.replication.factor=1\ntransaction.state.log.replication.factor=1\ntransaction.state.log.min.isr=1\nzookeeper.connect=zookeeper.kerberos-multi-node_default:2181\nzookeeper.set.acl=true\n\n# Kerberos / GSSAPI Authentication mechanism\nsasl.enabled.mechanisms=GSSAPI\nsasl.mechanism.inter.broker.protocol=GSSAPI\nsecurity.inter.broker.protocol=SASL_PLAINTEXT\nsasl.kerberos.service.name=kafka\nallow.everyone.if.no.acl.found=false\nsuper.users=User:admin;User:kafka\nauthorizer.class.name=kafka.security.auth.SimpleAclAuthorizer\n\n# metric reporter configuration with Kerberos\nmetric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter\nconfluent.metrics.reporter.bootstrap.servers=kafka:9093\nconfluent.metrics.reporter.sasl.mechanism=GSSAPI\nconfluent.metrics.reporter.security.protocol=SASL_PLAINTEXT\nconfluent.metrics.reporter.sasl.kerberos.service.name=kafka\nconfluent.metrics.reporter.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \\\n useKeyTab=true \\\n storeKey=true \\\n keyTab=\"/var/lib/secret/kafka.key\" \\\n principal=\"kafka@TEST.CONFLUENT.IO\";\nconfluent.metrics.reporter.topic.replicas=1\n"
},
{
"path": "kerberos-multi-node/kdc/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Installing Kerberos server, admin and client\nRUN yum install -y krb5-server krb5-libs \nRUN yum install -y krb5-workstation krb5-libs \n\n# 2. Configuring Kerberos and KDC\nCOPY krb5.conf /etc/krb5.conf\nRUN mkdir /var/log/kerberos\nRUN mkdir /etc/kdc\nRUN mkdir -p /var/kerberos/krb5kdc/ \nRUN ln -s /etc/krb5.conf /etc/kdc/krb5.conf\n\nEXPOSE 88\n\nRUN kdb5_util -P confluent -r TEST.CONFLUENT.IO create -s\n\nCMD /usr/sbin/krb5kdc -n\n"
},
{
"path": "kerberos-multi-node/kdc/krb5.conf",
"content": "[libdefaults]\n\tdefault_realm = TEST.CONFLUENT.IO\n ticket_lifetime = 24h\n renew_lifetime = 7d\n forwardable = true\n rdns = false\n dns_lookup_kdc = no\n dns_lookup_realm = no\n\n[realms]\n\tTEST.CONFLUENT.IO = {\n\t\tkdc = kdc\n\t\tadmin_server = kadmin\n\t}\n\n[domain_realm]\n\t.test.confluent.io = TEST.CONFLUENT.IO\n\ttest.confluent.io = TEST.CONFLUENT.IO\n kerberos_default = TEST.CONFLUENT.IO\n .kerberos_default = TEST.CONFLUENT.IO\n\n[logging]\n\tkdc = FILE:/var/log/kerberos/krb5kdc.log\n\tadmin_server = FILE:/var/log/kerberos/kadmin.log\n default = FILE:/var/log/kerberos/krb5lib.log\n"
},
{
"path": "kerberos-multi-node/up",
"content": "#!/bin/sh\n\n# Starting kerberos,\n# Avoiding starting up all services at the begining to generate the keytab first \n\ndocker-compose build\ndocker-compose up -d kdc\n\n# Create the required credential \ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey kafka_producer@TEST.CONFLUENT.IO\" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey kafka_consumer@TEST.CONFLUENT.IO\" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey kafka@TEST.CONFLUENT.IO\" > /dev/null\n\n# Create server keys \ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey zkservice/zookeeper.kerberos-multi-node_default@TEST.CONFLUENT.IO\" > /dev/null\n# princ for kafka0\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey kafka/kafka.kerberos-multi-node_default@TEST.CONFLUENT.IO\" > /dev/null\n# princ for kafka1\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey kafka/kafka1.kerberos-multi-node_default@TEST.CONFLUENT.IO\" > /dev/null\n\n# Create the keytab to use for Kafka\ndocker exec -ti kdc rm -f /var/lib/secret/kafka.key 2>&1 > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/kafka.key -glob zkservice/*\" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/kafka.key -glob kafka*\" > /dev/null\n\n# Starting zookeeper and kafka now that the keytab has been created with the required credentials and services\ndocker-compose up -d \n\n# Adding ACLs for consumer and producer user\ndocker-compose exec kafka bash -c \"kinit -k -t /var/lib/secret/kafka.key kafka && kafka-acls --authorizer-properties zookeeper.connect=zookeeper:2181 --add --allow-principal User:kafka_producer --producer --topic=*\"\ndocker-compose exec kafka bash -c \"kinit -k -t /var/lib/secret/kafka.key kafka && kafka-acls --authorizer-properties zookeeper.connect=zookeeper:2181 --add --allow-principal User:kafka_consumer --consumer --topic=* --group=*\"\n\necho \"Example configuration to access kafka:\"\necho \"-> docker-compose exec kafka bash -c 'kinit -k -t /var/lib/secret/kafka.key kafka_producer && kafka-console-producer --broker-list kafka:9093 --topic test --producer.config /etc/kafka/consumer.properties'\"\necho \"-> docker-compose exec kafka bash -c 'kinit -k -t /var/lib/secret/kafka.key kafka_consumer && kafka-console-consumer --bootstrap-server kafka:9093 --topic test --consumer.config /etc/kafka/consumer.properties --from-beginning'\"\n"
},
{
"path": "kerberos-multi-node/zookeeper/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/6.0/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-11-openjdk\nRUN yum install -y confluent-platform-2.12\n\n# 3. Configure zookeeper for Kerberos \nRUN yum install -y krb5-workstation krb5-libs \nCOPY zookeeper.properties /etc/kafka/zookeeper.properties\nCOPY zookeeper.sasl.jaas.config /etc/kafka/zookeeper_server_jaas.conf\n\nEXPOSE 2181\n\nCMD zookeeper-server-start /etc/kafka/zookeeper.properties \n"
},
{
"path": "kerberos-multi-node/zookeeper/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/6.0/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/6.0/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/6.0\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/6.0/archive.key\nenabled=1\n\n"
},
{
"path": "kerberos-multi-node/zookeeper/zookeeper.properties",
"content": "# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n# \n# http://www.apache.org/licenses/LICENSE-2.0\n# \n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# the directory where the snapshot is stored.\ndataDir=/var/lib/zookeeper\n# the port at which the clients will connect\nclientPort=2181\n# disable the per-ip limit on the number of connections since this is a non-production config\nmaxClientCnxns=0\nauthProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider\nzookeeper.allowSaslFailedClients=false\nrequireClientAuthScheme=sasl\nkerberos.removeHostFromPrincipal = true\nkerberos.removeRealmFromPrincipal = false\n"
},
{
"path": "kerberos-multi-node/zookeeper/zookeeper.sasl.jaas.config",
"content": "Server {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n\t\tuseTicketCache=false\n keyTab=\"/var/lib/secret/kafka.key\"\n principal=\"zkservice/zookeeper.kerberos-multi-node_default@TEST.CONFLUENT.IO\";\n};\n\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useTicketCache=true;\n};\n"
},
{
"path": "kerberos-multi-sasl/README.md",
"content": "# Kerberos configuration demo\n\nThis demo sets up a basic Kafka cluster secured with Kerberos authentication, and sets up some basic ACLs to demonstrate authorisation.\nThe documentation below introduces the relevant components you need to understand to set up Kerberos in a Linux / JVM environment.\n\n\n## Kerberos authentication process\n\nBefore configuring Kafka for Kerberos authentication, it is useful to understand the basics of Kerberos; the authentication process and some key terms.\n\n_A note on what Kerberos is and is not: Kerberos is a *network authentication protocol* which allows a client application to connect to a network service in a way that allows the components to mutually verify each other's identities._\n_It is put to good use in and integrated with network directory services, notably Windows Active Directory._\n_Here, Kerberos identities are bound to network accounts and access privileges and, in the case of Windows, the SSPI API supports single sign-on and privilege impersonation natively in the OS._\n_This is enabled by Kerberos, but Kerberos itself is not bound to such accounts and does not provide any such capability._\n\nWith that, let's work through the process for a client application making a connection into Kafka.\n\nKerberos involves three parties:\n\n- a Kerberos Client, in this case our client application.\n- a Kerberized Service, in this case Kafka.\n- the Kerberos **Key Distribution Center (KDC)**\n\nAn important point to understand in this process is that the Client and Service each shares their own cryptographic key with the KDC.\nBy using this key to encrypt/decrypt tokens passed over the network, two network systems can verify each other's identities.\nThe Client and Service trust that they have only shared their secret with the KDC and so any correctly signed token must have originated from the KDC.\nThis is crucial.\nDuring the Kerberos process the Client requests a token from the KDC _signed with the Service's key_ and presents this when making a connection.\nThe Service can then trust that the Client has valid credentials with the KDC and can be authenticated.\n\nOther information is shared during the process to enable integrity checking and protection against various spoofing attacks.\nFor example, each signed token is:\n\n* timestamped to bound the window for which it is valid\n* linked to a network IP so that it is valid only from a single host\n\nThe first stage is that the Client application must authenticate itself with the KDC by proving that the Client knows the private credentials relating to the Client's Kerberos **Principal**.\nThe Principal is a a unique identity in the form {primary}/{instance}@{REALM} (more on these later).\nThe KDC authenticates the client using their shared cryptographic key and results in the client receiving a **Ticket Granting Ticket (TGT)**.\nThis is a cryptographic token that the Client may now use to prove that it has recently authenticated with the KDC.\nThe TGT is timestamped and includes an expiry time, typically a day.\nThe TGT is cached by the client to avoid having to re-authenticate unnecessarily.\n\nNext, the Client wants to authenticate itself to the Kerberized Service.\nFor this to happen, the client must get a cryptographic token encrypted with the Kerberized Service's key - this token is a **Service Ticket** and is requested by the client from the KDC using the TGT and the requested service's principal name.\nIncluding the TGT in this request is sufficient to prove that the client has already authenticated with the KDC allowing the service ticket to be returned.\n\nHere is an important point to note - how does the client know the service principal name?\nVery simply, it builds the principal with:\n\n- {primary} = a client-side configured name for the service\n- {instance} = the network address used to connect to the service\n- {REALM} = the realm of the client and KDC.\n\nIn our example, our Client attempts to connect to the `kafka` Service on the host `kafka.kerberos-demo.local` in the realm `TEST.CONFLUENT.IO`.\nTherefore, the service must be configured with a Service Principal Name of `kafka/kafka.kerberos-demo.local@TEST.CONFLUENT.IO`.\n\nNow the Client can connect directly to the Kerberized Service, and include the Service Ticket.\nAs the Service ticket is signed with the Service principal's key, the Service can decrypt the token to authenticate the request.\n\nBased on the above, each connection in the cluster must be established with the following in place:\n\n* On the Kerberos Client:\n * A client principal and key to authenticate with the KDC, `{client name}@REALM`\n * This is the *User Principal Name*.\n * a configured name for the service to connect to, `{service name}`\n * the network address for the service, `{network address}`. \n* On the server:\n * a principal name & key in the form `{service name}/{network address}@REALM`.\n * This is the *Service Principal Name*.\n\nAs can be seen, the service principal must be constructed correctly to work.\nHowever, the `{client name}` format is not mandated in the same way and is not bound to a network address.\nOften the client name is a simple alphanumeric username, let's say 'john'.\nHowever, you may sometimes see a client principal such as 'john/admin'.\nIn this form, 'admin' is called an _instance_ of the 'john' principal and can be used by 'john' to run services on the system with different credentials and privileges from the main account.\nFrom the Kerberos perspective, the two principals are completely separate, but it can nonetheless be convenient to use this naming convention.\n\n\n# Technical Components\n## KDC\nThe KDC could be provided by MIT Kerberos, Windows Active Directory, Redhat Identity Manager and many others.\nIn this demo we use MIT kerberos.\n\n\n## Kerberos libraries and tools\n\nAll the hosts must include Kerberos libraries and a shared configuration (krb5.conf) in order to use and trust the same KDC.\n\n`kinit` is used to authenticate to the Kerberos server as principal, or if none is given, a system generated default (typically your login name at the default realm), and acquire a ticket granting ticket that can later be used to obtain tickets for other services.\n\n`klist` reads and displays the current tickets in the credential cache (also known as the ticket file).\n\n`kvno` acquires a service ticket for the specified Kerberos principals and prints out the key version numbers of each.\n\n`kadmin` is an admin utility for working with the Kerberos database.\n\nA common task when configuring for Kerberos is to build *keytab* files (short for Key Table).\nKeytabs are files containing one or more Kerberos principal/credential pairs.\nBy having these in a file, services can automatically authenticate with the KDC without prompting the user and it is common to build and distribute keytabs as part of a deployment.\nHowever, _as these files contain secret credentials, it is important to take care to protect against loss of these files_.\n\nSee [kerberos cheatsheet](../KerberosCheatsheet.md) for examples of using the Kerberos toolset.\n\n\n## Simple Authentication and Security Layer (SASL)\n\nSASL is a framework for authentication in network communications which in principle decouples authentication concerns from the application protocol.\n\nKafka and Zookeeper can use SASL as the authentication layer in communications (Mutual TLS being the notable alternative).\n\nWhen SASL has been enabled, you must further specify a SASL *mechanism* to use - the process and protocol to use when authenticating a connection.\nApplications must build support for each SASL mechanism - Kafka supports SCRAM(-SHA-256 | -SHA-512), PLAIN, OAUTHBEARER and GSSAPI.\n*GSSAPI is the SASL mechanism which implements Kerberos*.\n\n\n## Java Authentication and Authorization Services (JAAS)\n\nJAAS is a Java's integrated, pluggable security service and Kafka uses the JAAS APIs to implement SASL authentication.\nSASL authentication is configured using JAAS.\nKerberos is configured using the JAAS *LoginModule* `com.sun.security.auth.module.Krb5LoginModule`.\n\nJAAS may be configured in a couple of places:\n\n* By default it uses a .jaas file, a reference to which is passed in the `-Djava.security.auth.login.config=` JVM flag.\nEach jaas file includes multiple named stanzas, representing different login contexts.\n\n* An application can override this configuration and configure JAAS from application config.\nKafka configurations expose this option using properties `sasl.jaas.config`, which can variously be prefixed.\nThe value is the inline configuration for a single login context and, in Kafka, takes precedence over entries in a .jaas files.\n\nhttps://docs.oracle.com/javase/8/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html\n\n\nA Kerberos enabled Client or Service can be initiated in two ways:\n\n1. Use `kinit` to cache a TGT locally, and then launch the process with this shared cache.\n2. Configure a keytab to be used directly.\n\nConfiguration of the former is straight-forward as follows:\n\n```\nSomeLoginContext {\n com.sun.security.auth.module.Krb5LoginModule required\n useTicketCache = true;\n};\n```\n\nThe `useTicketCache = true` setting specifies that the TGT cache should be used.\n\nBy comparison, the latter approach has `useTicketCache = false` (the default) and then continues to specify details for using a keytab file:\n\n```\nSomeLoginContext {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/var/lib/secret/kafka1.key\"\n principal=\"kafka/kafka1.kerberos-demo.local@TEST.CONFLUENT.IO\";\n};\n```\n\nThe login context, as identified with `SomeLoginContext` above, can be used by a Client, a Service or both.\nFor Kafka, the names are defined in the application code as we will describe later.\n\n\n# Kerberizing Kafka\n\nTo fully understand the steps required to Kerberize Kafka, we should understand each Client → Service connection which we wish to configure.\nEach of these connections has a prototypical set of configurations required on the Client side and on the Service side.\n\nThe following are values you must decide upon at the cluster level:\n\n* `{kafka-kerberos-service-name}` - name for the Kerberized Kafka service.\nTypically `kafka` or `cp-kafka`.\n* `{zookeeper-kerberos-service-name}` - name for the Kerberized Zookeeper service.\nBy default this is `zookeeper`.\n* `{security-protocol}` - either `SASL_PLAINTEXT` of `SASL_SSL` if using in conjunction with TLS.\n\n## Service Configurations\n\nIn a single node Broker/Zookeeper environment there are just two Kerberized services running.\nWe will configure these first and then the clients.\n\n### Kafka Service\n\n* Broker JAAS:\n * Login Context: `KafkaServer`\n * Use *keytab* method.\n * Ensure that the principal is a correctly formed service principal for each node: `{kafka-kerberos-service-name}/{FQDN}@{realm}`.\n * Example: [kafka/kafka.sasl.jaas.config](kafka/kafka.sasl.jaas.config)\n* Broker Server Properties:\n * `sasl.enabled.mechanisms=GSSAPI` (more SASL mechanisms may be specified in a comma-separated list)\n * `sasl.kerberos.service.name={kafka-kerberos-service-name}`\n * `{listener_name}.{sasl_mechanism}.sasl.jaas.config` - jaas configuration on a per-listener basis.\n * Example: [kafka/server.properties](kafka/server.properties)\n\n### Zookeeper Service\n\n* Zookeeper JAAS:\n * Client API - Kerberize access to ZooKeeper data.\n * Login Context: `Server`.\n * Use *keytab* method.\n * Ensure that the principal is a correctly formed service principal for each node: `{zookeeper-kerberos-service-name}/{FQDN}@{realm}`.\n * Example: [zookeeper/zookeeper.sasl.jaas.config](zookeeper/zookeeper.sasl.jaas.config)\n* Zookeeper Properties:\n * authProvider.1 = org.apache.zookeeper.server.auth.SASLAuthenticationProvider\n * requireClientAuthScheme=sasl\n * Example: [zookeeper/zookeeper.properties](zookeeper/zookeeper.properties)\n\n\n## Client Configurations\n\n\n### Kafka Broker → Zookeeper Service\nBrokers connect to Zookeeper for cluster operations.\n\n* Broker JAAS:\n * Login Context: `Client`\n * Use *keytab* method.\n * *Ensure that the same principal is configured for use on each broker.*\n * Example: [kafka/kafka.sasl.jaas.config](kafka/kafka.sasl.jaas.config).\n* Broker JVM flags:\n * `-Dzookeeper.sasl.client.username={zookeeper-kerberos-service-name}` (OPTIONAL)\n\n\n### Client → Kafka Service\nClients connecting in to Kafka may be any of:\n\n* A Kafka producer\n* A Kafka consumer\n* A Kafka Admin client\n\nNote that many applications are a combination of many of these - notably Streams applications and Kafka Connect.\n\n* Client JAAS:\n * Login Context: `KafkaClient`\n * Can use *kinit* or *keytab* method.\n * Example: [client/client.sasl.jaas.config](client/client.sasl.jaas.config)\n* Client Properties:\n * `sasl.kerberos.service.name={kafka-kerberos-service-name}`\n * `security.protocol={security-protocol}`\n * `sasl.jaas.config` - jaas override.\n * Examples: [client/producer.properties](client/producer.properties), [client/consumer.properties](client/consumer.properties), [client/command.properties](client/command.properties)\n\n\n### Client → Zookeeper (Optional)\nHistorically, clients needed to connect directly to ZooKeeper for service discovery and admin operations.\nHowever, the new Kafka Admin API allows all this functionality via Client → Kafka Broker connection, so this direct connection should not be required.\n\n* JAAS:\n * LoginContext: `Client`\n * Can use *kinit* or *keytab* method.\n\n * JVM flags:\n * `-Dzookeeper.sasl.client.username={zookeeper-kerberos-service-name}` (OPTIONAL)\n\n### Confluent Metrics Reporter → Kafka Service (Optional)\n\nThe Confluent metrics reporter runs as a plugin within the Kafka broker, but from a Kerberos point of view is configured as a network client.\nThe configuration, including inline Jaas, is specified within the broker properties using a keytab:\n\n* `confluent.metrics.reporter.sasl.mechanism=GSSAPI`\n* `confluent.metrics.reporter.security.protocol={security-protocol}`\n* `confluent.metrics.reporter.sasl.kerberos.service.name={kafka-kerberos-service-name}`\n* `confluent.metrics.reporter.sasl.jaas.config={inline jaas configuration}`\n* Example: [kafka/server.properties](kafka/server.properties)\n\n# Authentication is not enough!\n\nThe steps above are sufficient to support Kerberos authenticated connections within the cluster.\nThis does not make your cluster secure though!\nThe demo also applies a minimal level of authorisation to prevent unauthenticated network access to the brokers and Zookeeper.\nThe following should be reviewed in the broker server properties:\n\n* `allow.everyone.if.no.acl.found=false` - default to no access.\n* `authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer` - enable the default authoriser for Kafka.\n* `zookeeper.set.acl=true` - when storing ACL data in Zookeeper, apply Zookeeper access controls so that only the Broker → Zookeeper client principal can read and modify the lists.\n* Example: [kafka/server.properties](kafka/server.properties)\n\n# Putting it into action\n\nIn this demo we configure:\n\n* A simple KDC to generate principals and keytabs.\n* A single node Zookeeper with a Kerberized data access API.\n* A single node Kafka broker with a Kerberized listener.\n* Set up ACLs allowing `kafka-console-producer` and `kafka-console-consumer` usage.\n\n_A basic knowledge of Docker is useful to follow the code, though only basic Docker techniques are used to keep the code readable._\n_Each node is built using a `Dockerfile` into which configuration values are hard-coded, and the services are brought up using `docker-compose`._\n_Kerberos keytabs and the krb5.conf file are shared amongst all nodes on the cluster using a shared Docker volume._\n\nThe demo is run using the [up](up) script, which orchestrates the following process:\n\n* Builds and starts the KDC.\nAll nodes are joined to the KDC's realm by sharing `krb5.conf` amongst all nodes.\n* Generates Kerberos principals and keytabs, sharing these on the shared Docker volume.\n* Builds and starts Zookeeper, Kafka broker and Client.\n* Uses the `admin` super user to configure ACLs for the `producer` and `consumer` users.\n* Prints example usage to connect into Kafka with a Kerberos principal.\nThis is actually executed via the `client` node.\n\n# Next up\n\n* Extending Kerberos configuration to a full cluster (coming soon)\n* Hardening access controls\n\n\n# References\n\n* https://www.youtube.com/watch?v=KD2Q-2ToloE Video overview of Kerberos authentication process.\n"
},
{
"path": "kerberos-multi-sasl/client/Dockerfile",
"content": "FROM centos:centos7\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.4/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install confluent kafka tools:\nRUN yum install -y java-11-openjdk\nRUN yum install -y confluent-platform-2.12\n\n# 3. Install Kerberos libaries\nRUN yum install -y krb5-workstation krb5-libs\n\n# 4. Copy in required settings for client access to Kafka\nCOPY consumer.properties /etc/kafka/consumer.properties\nCOPY producer.properties /etc/kafka/producer.properties\nCOPY command.properties /etc/kafka/command.properties\nCOPY scram.properties /etc/kafka/scram.properties\nCOPY client.sasl.jaas.config /etc/kafka/client_jaas.conf\n\nENV KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf\n\nCMD sleep infinity\n"
},
{
"path": "kerberos-multi-sasl/client/client.sasl.jaas.config",
"content": "/*\n* Credentials to use when connecting to ZooKeeper directly.\n*\n* Whenever possible you should use the Kafka AdminClient API instead of ZooKeeper.\n*/\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useTicketCache=true;\n};\n\n\n/*\n* Credentials to connect to Kafka.\n*/\nKafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required \n useTicketCache=true;\n};\n"
},
{
"path": "kerberos-multi-sasl/client/command.properties",
"content": "bootstrap.servers=kafka:9093\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \\\n\t\t\t\t\t\t\t\t serviceName=kafka \\\n\t\t\t\t\t\t\t\t useTicketCache=true;\n"
},
{
"path": "kerberos-multi-sasl/client/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.4/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.4\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n"
},
{
"path": "kerberos-multi-sasl/client/consumer.properties",
"content": "bootstrap.servers=kafka:9093\nsecurity.protocol=SASL_PLAINTEXT\nsasl.kerberos.service.name=kafka\nsasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \\\n\t\t\t\t\t\t\t\t useTicketCache=true;\n"
},
{
"path": "kerberos-multi-sasl/client/producer.properties",
"content": "bootstrap.servers=kafka:9093\nsecurity.protocol=SASL_PLAINTEXT\nsasl.kerberos.service.name=kafka\nsasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \\\n\t\t\t\t\t\t\t\t useTicketCache=true;\n"
},
{
"path": "kerberos-multi-sasl/client/scram.properties",
"content": "sasl.mechanism=SCRAM-SHA-512\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"kafka\" \\\n password=\"kafka\";\n"
},
{
"path": "kerberos-multi-sasl/docker-compose.yml",
"content": "version: '3.5'\nservices:\n kdc:\n hostname: kdc.kerberos-demo.local\n #domainname: kerberos_default\n build: kdc/\n container_name: kdc\n volumes:\n - secret:/var/lib/secret\n - ./kdc/krb5.conf:/etc/kdc/krb5.conf\n\n zookeeper:\n build: zookeeper/\n container_name: zookeeper\n hostname: zookeeper.kerberos-demo.local\n #domainname: kerberos_default\n depends_on:\n - kdc\n # Required to wait for the keytab to get generated\n restart: on-failure\n volumes:\n - secret:/var/lib/secret\n - ./kdc/krb5.conf:/etc/krb5.conf\n\n kafka:\n build: kafka/\n container_name: kafka\n hostname: kafka.kerberos-demo.local\n #domainname: kerberos_default\n depends_on:\n - zookeeper\n - kdc\n # Required to wait for the keytab to get generated\n restart: on-failure\n volumes:\n - secret:/var/lib/secret\n - ./kdc/krb5.conf:/etc/krb5.conf\n\n client:\n build: client/\n container_name: client\n hostname: client.kerberos-demo.local\n #domainname: kerberos_default\n depends_on:\n - kafka\n - kdc\n # Required to wait for the keytab to get generated\n volumes:\n - secret:/var/lib/secret\n - ./kdc/krb5.conf:/etc/krb5.conf\n\nvolumes:\n secret: {}\n\nnetworks:\n default:\n name: kerberos-demo.local\n"
},
{
"path": "kerberos-multi-sasl/kafka/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.4/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-11-openjdk\nRUN yum install -y confluent-platform-2.12\nRUN yum install -y confluent-control-center\n\n# 3. Configure Kafka for Kerberos\nRUN yum install -y krb5-workstation krb5-libs\nCOPY server.properties /etc/kafka/server.properties\nCOPY kafka.sasl.jaas.config /etc/kafka/kafka_server_jaas.conf\n\nEXPOSE 9093 \n\nENV KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf \n\nCMD kafka-server-start /etc/kafka/server.properties\n"
},
{
"path": "kerberos-multi-sasl/kafka/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.4/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.4\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n"
},
{
"path": "kerberos-multi-sasl/kafka/kafka.sasl.jaas.config",
"content": "/*\n* The service principal\n*/\n/*\nKafkaServer {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/var/lib/secret/kafka.key\"\n principal=\"kafka/kafka.kerberos-demo.local@TEST.CONFLUENT.IO\";\n};\n*/\n\n/*\n* Zookeeper client principal\n*/\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n\t\tuseTicketCache=false\n keyTab=\"/var/lib/secret/zookeeper-client.key\"\n principal=\"zkclient@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kerberos-multi-sasl/kafka/server.properties",
"content": "# Basic broker and listener configuration\nbroker.id=0\nlisteners=SASL_PLAINTEXT://kafka.kerberos-demo.local:9093\nzookeeper.connect=zookeeper.kerberos-demo.local:2181\nlog.dirs=/var/lib/kafka\n\noffsets.topic.replication.factor=1\ntransaction.state.log.replication.factor=1\ntransaction.state.log.min.isr=1\nnum.partitions=12\n\n\n# Kerberos / GSSAPI Authentication mechanism\nsasl.enabled.mechanisms=SCRAM-SHA-512,GSSAPI\nsasl.kerberos.service.name=kafka\nlistener.name.sasl_plaintext.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \\\n useKeyTab=true \\\n storeKey=true \\\n keyTab=\"/var/lib/secret/kafka.key\" \\\n principal=\"kafka/kafka.kerberos-demo.local@TEST.CONFLUENT.IO\";\nlistener.name.sasl_plaintext.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"kafka\" \\\n password=\"kafka\";\n# Configure replication to require Kerberos:\nsasl.mechanism.inter.broker.protocol=GSSAPI\nsecurity.inter.broker.protocol=SASL_PLAINTEXT\n\n\n# Authorization config:\nauthorizer.class.name=kafka.security.auth.SimpleAclAuthorizer\nzookeeper.set.acl=true\nallow.everyone.if.no.acl.found=false\nsuper.users=User:admin;User:kafka\n\n\n# Demonstrate setting up the Confluent Metrics Reporter with required *client* credentials\nmetric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter\nconfluent.metrics.reporter.bootstrap.servers=kafka:9093\nconfluent.metrics.reporter.sasl.mechanism=GSSAPI\nconfluent.metrics.reporter.security.protocol=SASL_PLAINTEXT\nconfluent.metrics.reporter.sasl.kerberos.service.name=kafka\nconfluent.metrics.reporter.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \\\n useKeyTab=true \\\n storeKey=true \\\n keyTab=\"/var/lib/secret/kafka-admin.key\" \\\n principal=\"admin/for-kafka@TEST.CONFLUENT.IO\";\n\nconfluent.metrics.reporter.topic.replicas=1\nconfluent.support.metrics.enable=false\nconfluent.support.customer.id=anonymous\n"
},
{
"path": "kerberos-multi-sasl/kdc/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Installing Kerberos server, admin and client\nRUN yum install -y krb5-server krb5-libs \nRUN yum install -y krb5-workstation krb5-libs \n\n# 2. Configuring Kerberos and KDC\nCOPY krb5.conf /etc/krb5.conf\nRUN mkdir /var/log/kerberos\nRUN mkdir /etc/kdc\nRUN mkdir -p /var/kerberos/krb5kdc/ \nRUN ln -s /etc/krb5.conf /etc/kdc/krb5.conf\n\nEXPOSE 88\n\nRUN kdb5_util -P confluent -r TEST.CONFLUENT.IO create -s\n\nCMD /usr/sbin/krb5kdc -n\n"
},
{
"path": "kerberos-multi-sasl/kdc/krb5.conf",
"content": "[libdefaults]\n\tdefault_realm = TEST.CONFLUENT.IO\n ticket_lifetime = 24h\n renew_lifetime = 7d\n forwardable = true\n rdns = false\n dns_lookup_kdc = no\n dns_lookup_realm = no\n\n[realms]\n\tTEST.CONFLUENT.IO = {\n\t\tkdc = kdc\n\t\tadmin_server = kadmin\n\t}\n\n[domain_realm]\n\t.test.confluent.io = TEST.CONFLUENT.IO\n\ttest.confluent.io = TEST.CONFLUENT.IO\n kerberos-demo.local = TEST.CONFLUENT.IO\n .kerberos-demo.local = TEST.CONFLUENT.IO\n\n[logging]\n\tkdc = FILE:/var/log/kerberos/krb5kdc.log\n\tadmin_server = FILE:/var/log/kerberos/kadmin.log\n default = FILE:/var/log/kerberos/krb5lib.log\n"
},
{
"path": "kerberos-multi-sasl/up",
"content": "#!/bin/sh\nset -e\n\n# Starting kerberos,\n# Avoiding starting up all services at the begining to generate the keytab first\n\ndocker-compose build\ndocker-compose up -d kdc\n\n### Create the required identities:\n# Kafka service principal:\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey kafka/kafka.kerberos-demo.local@TEST.CONFLUENT.IO\" > /dev/null\n\n# Zookeeper service principal:\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey zookeeper/zookeeper.kerberos-demo.local@TEST.CONFLUENT.IO\" > /dev/null\n\n# Create a principal with which to connect to Zookeeper from brokers - NB use the same credential on all brokers!\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey zkclient@TEST.CONFLUENT.IO\" > /dev/null\n\n# Create client principals to connect in to the cluster:\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey kafka_producer@TEST.CONFLUENT.IO\" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey kafka_producer/instance_demo@TEST.CONFLUENT.IO\" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey kafka_consumer@TEST.CONFLUENT.IO\" > /dev/null\n\n# Create an admin principal for the cluster, which we'll use to setup ACLs.\n# Look after this - its also declared a super user in broker config.\ndocker exec -ti kdc kadmin.local -w password -q \"add_principal -randkey admin/for-kafka@TEST.CONFLUENT.IO\" > /dev/null\n\n# Create keytabs to use for Kafka\ndocker exec -ti kdc rm -f /var/lib/secret/kafka.key 2>&1 > /dev/null\ndocker exec -ti kdc rm -f /var/lib/secret/zookeeper.key 2>&1 > /dev/null\ndocker exec -ti kdc rm -f /var/lib/secret/zookeeper-client.key 2>&1 > /dev/null\ndocker exec -ti kdc rm -f /var/lib/secret/kafka-client.key 2>&1 > /dev/null\ndocker exec -ti kdc rm -f /var/lib/secret/kafka-admin.key 2>&1 > /dev/null\n\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/kafka.key -norandkey kafka/kafka.kerberos-demo.local@TEST.CONFLUENT.IO \" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/zookeeper.key -norandkey zookeeper/zookeeper.kerberos-demo.local@TEST.CONFLUENT.IO \" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/zookeeper-client.key -norandkey zkclient@TEST.CONFLUENT.IO \" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/kafka-client.key -norandkey kafka_producer@TEST.CONFLUENT.IO \" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/kafka-client.key -norandkey kafka_producer/instance_demo@TEST.CONFLUENT.IO \" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/kafka-client.key -norandkey kafka_consumer@TEST.CONFLUENT.IO \" > /dev/null\ndocker exec -ti kdc kadmin.local -w password -q \"ktadd -k /var/lib/secret/kafka-admin.key -norandkey admin/for-kafka@TEST.CONFLUENT.IO \" > /dev/null\n\n# Starting zookeeper and kafka now that the keytab has been created with the required credentials and services\ndocker-compose up -d\n\n# Adding SCRAM user \n\ndocker-compose exec kafka kafka-configs --zookeeper zookeeper:2181 --alter --add-config 'SCRAM-SHA-256=[password=kafka],SCRAM-SHA-512=[password=kafka]' --entity-type users --entity-name kafka\n\n# Adding ACLs for consumer and producer user:\ndocker exec client bash -c \"kinit -k -t /var/lib/secret/kafka-admin.key admin/for-kafka && kafka-acls --bootstrap-server kafka:9093 --command-config /etc/kafka/command.properties --add --allow-principal User:kafka_producer --producer --topic=*\"\ndocker exec client bash -c \"kinit -k -t /var/lib/secret/kafka-admin.key admin/for-kafka && kafka-acls --bootstrap-server kafka:9093 --command-config /etc/kafka/command.properties --add --allow-principal User:kafka_consumer --consumer --topic=* --group=*\"\n\n\n# Output example usage:\necho \"Example configuration to access kafka:\"\necho \"-> docker-compose exec client bash -c 'kinit -k -t /var/lib/secret/kafka-client.key kafka_producer && kafka-console-producer --broker-list kafka:9093 --topic test --producer.config /etc/kafka/producer.properties'\"\necho \"-> docker-compose exec client bash -c 'kafka-console-producer --broker-list kafka:9093 --topic test --producer.config /etc/kafka/scram.properties'\"\necho \"-> docker-compose exec client bash -c 'kinit -k -t /var/lib/secret/kafka-client.key kafka_consumer && kafka-console-consumer --bootstrap-server kafka:9093 --topic test --consumer.config /etc/kafka/consumer.properties --from-beginning'\"\n"
},
{
"path": "kerberos-multi-sasl/zookeeper/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.4/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-11-openjdk\nRUN yum install -y confluent-platform-2.12\n\n# 3. Configure zookeeper for Kerberos\nRUN yum install -y krb5-workstation krb5-libs\nCOPY zookeeper.properties /etc/kafka/zookeeper.properties\nCOPY zookeeper.sasl.jaas.config /etc/kafka/zookeeper_server_jaas.conf\n\nEXPOSE 2181\n\nENV KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/zookeeper_server_jaas.conf\n\nCMD zookeeper-server-start /etc/kafka/zookeeper.properties\n"
},
{
"path": "kerberos-multi-sasl/zookeeper/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.4/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.4\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n"
},
{
"path": "kerberos-multi-sasl/zookeeper/zookeeper.properties",
"content": "dataDir=/var/lib/zookeeper\nclientPort=2181\nmaxClientCnxns=0\nauthProvider.1 = org.apache.zookeeper.server.auth.SASLAuthenticationProvider\nzookeeper.allowSaslFailedClients=false\nrequireClientAuthScheme=sasl\n"
},
{
"path": "kerberos-multi-sasl/zookeeper/zookeeper.sasl.jaas.config",
"content": "Server {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n keyTab=\"/var/lib/secret/zookeeper.key\"\n storeKey=true\n useTicketCache=false\n principal=\"zookeeper/zookeeper.kerberos-demo.local@TEST.CONFLUENT.IO\";\n};\n\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useTicketCache=true;\n};\n"
},
{
"path": "kraft/none/docker-compose.yml",
"content": "---\nversion: '3'\nservices:\n kafka-controller-1:\n build: ./image/kafka-images/kafka/\n hostname: kafka-controller-1\n container_name: kafka-controller-1\n environment:\n KAFKA_LISTENERS: CONTROLLER://:9092\n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,OUTSIDE:PLAINTEXT\n KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0\n KAFKA_PROCESS_ROLES: controller\n KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-controller-1:9092,2@kafka-controller-2:9092,3@kafka-controller-3:9092\n KAFKA_NODE_ID: 1\n KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER\n ports:\n - 9093:9093\n kafka-controller-2:\n build: ./image/kafka-images/kafka/\n hostname: kafka-controller-2\n container_name: kafka-controller-2\n environment:\n KAFKA_LISTENERS: CONTROLLER://:9092\n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,OUTSIDE:PLAINTEXT\n KAFKA_PROCESS_ROLES: controller\n KAFKA_NODE_ID: 2\n KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-controller-1:9092,2@kafka-controller-2:9092,3@kafka-controller-3:9092\n KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER\n ports:\n - 9094:9094\n\n kafka-controller-3:\n build: ./image/kafka-images/kafka/\n hostname: kafka-controller-3\n container_name: kafka-controller-3\n environment:\n KAFKA_LISTENERS: CONTROLLER://:9092\n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,OUTSIDE:PLAINTEXT\n KAFKA_PROCESS_ROLES: controller\n KAFKA_NODE_ID: 3\n KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-controller-1:9092,2@kafka-controller-2:9092,3@kafka-controller-3:9092\n KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER\n ports:\n - 9095:9095\n\n kafka-1:\n build: ./image/kafka-images/kafka/\n hostname: kafka-1\n container_name: kafka-1\n environment:\n KAFKA_BROKER_ID: 11\n KAFKA_LISTENERS: INTERNAL://:9092,OUTSIDE://:9091\n KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-1:9092,OUTSIDE://localhost:9091\n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT,CONTROLLER:PLAINTEXT\n KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL\n KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0\n KAFKA_JMX_PORT: 9999\n KAFKA_JMX_HOSTNAME: kafka-1\n KAFKA_PROCESS_ROLES: broker\n KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-controller-1:9092,2@kafka-controller-2:9092,3@kafka-controller-3:9092\n KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER\n ports:\n - 9091:9091\n\n kafka-2:\n build: ./image/kafka-images/kafka/\n hostname: kafka-2\n container_name: kafka-2\n environment:\n KAFKA_BROKER_ID: 12\n KAFKA_LISTENERS: INTERNAL://:9092,OUTSIDE://:9090\n KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-1:9092,OUTSIDE://localhost:9090\n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT,CONTROLLER:PLAINTEXT\n KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL\n KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0\n KAFKA_JMX_PORT: 9999\n KAFKA_JMX_HOSTNAME: kafka-2\n KAFKA_PROCESS_ROLES: broker\n KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-controller-1:9092,2@kafka-controller-2:9092,3@kafka-controller-3:9092\n KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER\n ports:\n - 9090:9090\n\n kafka-3:\n build: ./image/kafka-images/kafka/\n hostname: kafka-3\n container_name: kafka-3\n environment:\n KAFKA_BROKER_ID: 13\n KAFKA_LISTENERS: INTERNAL://:9092,OUTSIDE://:9089\n KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-1:9092,OUTSIDE://localhost:9089\n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT,CONTROLLER:PLAINTEXT\n KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL\n KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0\n KAFKA_JMX_PORT: 9999\n KAFKA_JMX_HOSTNAME: kafka-3\n KAFKA_PROCESS_ROLES: broker\n KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-controller-1:9092,2@kafka-controller-2:9092,3@kafka-controller-3:9092\n KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER\n ports:\n - 9089:9089\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/Dockerfile",
"content": "#\n# Copyright 2019 Confluent Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nARG DOCKER_UPSTREAM_REGISTRY\nARG DOCKER_UPSTREAM_TAG=7.2.1\n\nFROM ${DOCKER_UPSTREAM_REGISTRY}confluentinc/cp-base-new:${DOCKER_UPSTREAM_TAG}\n\nARG PROJECT_VERSION\nARG ARTIFACT_ID\nARG GIT_COMMIT\n\nLABEL maintainer=\"partner-support@confluent.io\"\nLABEL vendor=\"Confluent\"\nLABEL version=$GIT_COMMIT\nLABEL release=$PROJECT_VERSION\nLABEL name=$ARTIFACT_ID\nLABEL summary=\"Confluent platform Kafka.\"\nLABEL io.confluent.docker=true\nLABEL io.confluent.docker.git.id=$GIT_COMMIT\nARG BUILD_NUMBER=-1\nLABEL io.confluent.docker.build.number=$BUILD_NUMBER\nLABEL io.confluent.docker.git.repo=\"confluentinc/kafka-images\"\n\nARG CONFLUENT_VERSION=7.2.1\nARG CONFLUENT_PACKAGES_REPO=https://packages.confluent.io/rpm/7.2\nARG CONFLUENT_PLATFORM_LABEL=7.2.1\n\nENV COMPONENT=kafka\n\n# primary\nEXPOSE 9092\n\nUSER root\n\nRUN echo \"===> Installing ${COMPONENT}...\" \\\n && echo \"===> Adding confluent repository...${CONFLUENT_PACKAGES_REPO}\" \\\n && rpm --import ${CONFLUENT_PACKAGES_REPO}/archive.key \\\n && printf \"[Confluent.dist] \\n\\\nname=Confluent repository (dist) \\n\\\nbaseurl=${CONFLUENT_PACKAGES_REPO}/\\$releasever \\n\\\ngpgcheck=1 \\n\\\ngpgkey=${CONFLUENT_PACKAGES_REPO}/archive.key \\n\\\nenabled=1 \\n\\\n\\n\\\n[Confluent] \\n\\\nname=Confluent repository \\n\\\nbaseurl=${CONFLUENT_PACKAGES_REPO}/ \\n\\\ngpgcheck=1 \\n\\\ngpgkey=${CONFLUENT_PACKAGES_REPO}/archive.key \\n\\\nenabled=1 \" > /etc/yum.repos.d/confluent.repo \\\n && yum install -y confluent-kafka-${CONFLUENT_VERSION} \\\n && echo \"===> clean up ...\" \\\n && yum clean all \\\n && rm -rf /tmp/* /etc/yum.repos.d/confluent.repo \\\n && echo \"===> Setting up ${COMPONENT} dirs\" \\\n && mkdir -p /var/lib/${COMPONENT}/data /etc/${COMPONENT}/secrets \\\n && chown appuser:root -R /etc/kafka /var/log/kafka /var/log/confluent /var/lib/kafka /var/lib/zookeeper /etc/${COMPONENT}/secrets /var/lib/${COMPONENT} /etc/${COMPONENT} \\\n && chmod -R ug+w /etc/kafka /var/log/kafka /var/log/confluent /var/lib/kafka /var/lib/zookeeper /var/lib/${COMPONENT} /etc/${COMPONENT}/secrets /etc/${COMPONENT}\n\nVOLUME [\"/var/lib/${COMPONENT}/data\", \"/etc/${COMPONENT}/secrets\"]\n\nCOPY --chown=appuser:appuser include/etc/confluent/docker /etc/confluent/docker\n\nUSER appuser\n\nCMD [\"/etc/confluent/docker/run\"]\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/Dockerfile.ubi8",
"content": "#\n# Copyright 2019 Confluent Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nARG DOCKER_UPSTREAM_REGISTRY\nARG DOCKER_UPSTREAM_TAG=7.2.1\n\nFROM ${DOCKER_UPSTREAM_REGISTRY}confluentinc/cp-base-new:${DOCKER_UPSTREAM_TAG}\n\nARG PROJECT_VERSION\nARG ARTIFACT_ID\nARG GIT_COMMIT\n\nLABEL maintainer=\"partner-support@confluent.io\"\nLABEL vendor=\"Confluent\"\nLABEL version=$GIT_COMMIT\nLABEL release=$PROJECT_VERSION\nLABEL name=$ARTIFACT_ID\nLABEL summary=\"Confluent platform Kafka.\"\nLABEL io.confluent.docker=true\nLABEL io.confluent.docker.git.id=$GIT_COMMIT\nARG BUILD_NUMBER=-1\nLABEL io.confluent.docker.build.number=$BUILD_NUMBER\nLABEL io.confluent.docker.git.repo=\"confluentinc/kafka-images\"\n\nARG CONFLUENT_VERSION=7.2.1\nARG CONFLUENT_PACKAGES_REPO=https://packages.confluent.io/rpm/7.2\nARG CONFLUENT_PLATFORM_LABEL=7.2.1\n\nENV COMPONENT=kafka\n\n# primary\nEXPOSE 9092\n\nUSER root\n\nRUN echo \"===> Installing ${COMPONENT}...\" \\\n && echo \"===> Adding confluent repository...${CONFLUENT_PACKAGES_REPO}\" \\\n && rpm --import ${CONFLUENT_PACKAGES_REPO}/archive.key \\\n && printf \"[Confluent.dist] \\n\\\nname=Confluent repository (dist) \\n\\\nbaseurl=${CONFLUENT_PACKAGES_REPO}/\\$releasever \\n\\\ngpgcheck=1 \\n\\\ngpgkey=${CONFLUENT_PACKAGES_REPO}/archive.key \\n\\\nenabled=1 \\n\\\n\\n\\\n[Confluent] \\n\\\nname=Confluent repository \\n\\\nbaseurl=${CONFLUENT_PACKAGES_REPO}/ \\n\\\ngpgcheck=1 \\n\\\ngpgkey=${CONFLUENT_PACKAGES_REPO}/archive.key \\n\\\nenabled=1 \" > /etc/yum.repos.d/confluent.repo \\\n && yum install -y confluent-kafka-${CONFLUENT_VERSION} \\\n && echo \"===> clean up ...\" \\\n && yum clean all \\\n && rm -rf /tmp/* /etc/yum.repos.d/confluent.repo \\\n && echo \"===> Setting up ${COMPONENT} dirs\" \\\n && mkdir -p /var/lib/${COMPONENT}/data /etc/${COMPONENT}/secrets \\\n && chown appuser:root -R /etc/kafka /var/log/kafka /var/log/confluent /var/lib/kafka /var/lib/zookeeper /etc/${COMPONENT}/secrets /var/lib/${COMPONENT} /etc/${COMPONENT} \\\n && chmod -R ug+w /etc/kafka /var/log/kafka /var/log/confluent /var/lib/kafka /var/lib/zookeeper /var/lib/${COMPONENT} /etc/${COMPONENT}/secrets /etc/${COMPONENT}\n\nVOLUME [\"/var/lib/${COMPONENT}/data\", \"/etc/${COMPONENT}/secrets\"]\n\nCOPY --chown=appuser:appuser include/etc/confluent/docker /etc/confluent/docker\n\nUSER appuser\n\nCMD [\"/etc/confluent/docker/run\"]\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/README.md",
"content": "# Confluent Community Docker Image for Apache Kafka\n\nDocker image for deploying and running the Community Version of Kafka packaged with the Confluent Community download. Please see the [cp-server](https://hub.docker.com/r/confluentinc/cp-server) image for additional commercial features that are only part of [Confluent Server](https://docs.confluent.io/platform/current/installation/available_packages.html#confluent-server).\n\n## Using the image\n\n* [Notes on using the image](https://docs.confluent.io/platform/current/installation/docker/installation.html) \n* [Configuration Reference](https://docs.confluent.io/platform/current/installation/docker/config-reference.html#confluent-ak-configuration)\n\n## Resources\n\n* [Docker Quick Start for Apache Kafka using Confluent Platform](https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html#ce-docker-quickstart)\n\n* [Learn Kafka](https://developer.confluent.io/learn-kafka)\n\n* [Confluent Developer](https://developer.confluent.io): blogs, tutorials, videos, and podcasts for learning all about Apache Kafka and Confluent Platform\n\n* [confluentinc/cp-demo](https://github.com/confluentinc/cp-demo): GitHub demo that you can run locally. The demo uses this Docker image to showcase Confluent Server in a secured, end-to-end event streaming platform. It has an accompanying playbook that shows users how to use Confluent Control Center to manage and monitor Kafka connect, Schema Registry, REST Proxy, KSQL, and Kafka Streams.\n\n* [confluentinc/examples](https://github.com/confluentinc/examples): additional curated examples in GitHub that you can run locally.\n\n## Contribute\n\nStart by reading our guidelines on contributing to this project found here.\n\n* [Source Code](https://github.com/confluentinc/kafka-images)\n* [Issue Tracker](https://github.com/confluentinc/kafka-images/issues)\n\n## License\n\nThis Docker image is licensed under the Apache 2 license. For more information on the licenses for each of the individual Confluent Platform components packaged in this image, please refer to the respective [Confluent Platform documentation](https://docs.confluent.io/platform/current/installation/docker/image-reference.html)."
},
{
"path": "kraft/none/image/kafka-images/kafka/include/etc/confluent/docker/configure",
"content": "#!/usr/bin/env bash\n#\n# Copyright 2016 Confluent Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n. /etc/confluent/docker/bash-config\n\ndub path /etc/kafka/ writable\n\nif [[ -z \"${KAFKA_LOG_DIRS-}\" ]]\nthen\n export KAFKA_LOG_DIRS\n KAFKA_LOG_DIRS=\"/var/lib/kafka/data\"\nfi\n\n# advertised.host, advertised.port, host and port are deprecated. Exit if these properties are set.\nif [[ -n \"${KAFKA_ADVERTISED_PORT-}\" ]]\nthen\n echo \"advertised.port is deprecated. Please use KAFKA_ADVERTISED_LISTENERS instead.\"\n exit 1\nfi\n\nif [[ -n \"${KAFKA_ADVERTISED_HOST-}\" ]]\nthen\n echo \"advertised.host is deprecated. Please use KAFKA_ADVERTISED_LISTENERS instead.\"\n exit 1\nfi\n\nif [[ -n \"${KAFKA_HOST-}\" ]]\nthen\n echo \"host is deprecated. Please use KAFKA_ADVERTISED_LISTENERS instead.\"\n exit 1\nfi\n\nif [[ -n \"${KAFKA_PORT-}\" ]]\nthen\n echo \"port is deprecated. Please use KAFKA_ADVERTISED_LISTENERS instead.\"\n exit 1\nfi\n\ndub template \"/etc/confluent/docker/${COMPONENT}.properties.template\" \"/etc/${COMPONENT}/${COMPONENT}.properties\"\ndub template \"/etc/confluent/docker/log4j.properties.template\" \"/etc/${COMPONENT}/log4j.properties\"\ndub template \"/etc/confluent/docker/tools-log4j.properties.template\" \"/etc/${COMPONENT}/tools-log4j.properties\"\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/include/etc/confluent/docker/ensure",
"content": "#!/usr/bin/env bash\n#\n# Copyright 2020 Confluent Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n. /etc/confluent/docker/bash-config\n\nexport KAFKA_DATA_DIRS=${KAFKA_DATA_DIRS:-\"/var/lib/kafka/data\"}\necho \"===> Check if $KAFKA_DATA_DIRS is writable ...\"\ndub path \"$KAFKA_DATA_DIRS\" writable\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/include/etc/confluent/docker/kafka.properties.template",
"content": "{% set excluded_props = ['KAFKA_VERSION',\n 'KAFKA_HEAP_OPTS'\n 'KAFKA_LOG4J_OPTS',\n 'KAFKA_OPTS',\n 'KAFKA_JMX_OPTS',\n 'KAFKA_JVM_PERFORMANCE_OPTS',\n 'KAFKA_GC_LOG_OPTS',\n 'KAFKA_LOG4J_ROOT_LOGLEVEL',\n 'KAFKA_LOG4J_LOGGERS',\n 'KAFKA_TOOLS_LOG4J_LOGLEVEL',\n 'KAFKA_ZOOKEEPER_CLIENT_CNXN_SOCKET']\n-%}\n\n{# properties that don't fit the standard format #}\n{% set other_props = {\n 'KAFKA_ZOOKEEPER_CLIENT_CNXN_SOCKET' : 'zookeeper.clientCnxnSocket'\n } -%}\n\n{% set kafka_props = env_to_props('KAFKA_', '', exclude=excluded_props) -%}\n{% for name, value in kafka_props.items() -%}\n{{name}}={{value}}\n{% endfor -%}\n\n{% for k, property in other_props.items() -%}\n{% if env.get(k) != None -%}\n{{property}}={{env[k]}}\n{% endif -%}\n{% endfor -%}\n\n{% set confluent_support_props = env_to_props('CONFLUENT_SUPPORT_', 'confluent.support.') -%}\n{% for name, value in confluent_support_props.items() -%}\n{{name}}={{value}}\n{% endfor -%}\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/include/etc/confluent/docker/launch",
"content": "#!/usr/bin/env bash\n#\n# Copyright 2016 Confluent Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# Override this section from the script to include the com.sun.management.jmxremote.rmi.port property.\nif [ -z \"$KAFKA_JMX_OPTS\" ]; then\n export KAFKA_JMX_OPTS=\"-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false \"\nfi\n\n# The JMX client needs to be able to connect to java.rmi.server.hostname.\n# The default for bridged n/w is the bridged IP so you will only be able to connect from another docker container.\n# For host n/w, this is the IP that the hostname on the host resolves to.\n\n# If you have more that one n/w configured, hostname -i gives you all the IPs,\n# the default is to pick the first IP (or network).\nexport KAFKA_JMX_HOSTNAME=${KAFKA_JMX_HOSTNAME:-$(hostname -i | cut -d\" \" -f1)}\n\nif [ \"$KAFKA_JMX_PORT\" ]; then\n # This ensures that the \"if\" section for JMX_PORT in kafka launch script does not trigger.\n export JMX_PORT=$KAFKA_JMX_PORT\n export KAFKA_JMX_OPTS=\"$KAFKA_JMX_OPTS -Djava.rmi.server.hostname=$KAFKA_JMX_HOSTNAME -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT -Dcom.sun.management.jmxremote.port=$JMX_PORT\"\nfi\n\necho \"===> Launching ${COMPONENT} ... \"\ncat /etc/\"${COMPONENT}\"/\"${COMPONENT}\".properties\nkafka-storage format -c \"/etc/${COMPONENT}/${COMPONENT}.properties\" -t Uo9ADuclRIG1r2kcxn4Alw\nexec \"${COMPONENT}\"-server-start /etc/\"${COMPONENT}\"/\"${COMPONENT}\".properties\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/include/etc/confluent/docker/log4j.properties.template",
"content": "\nlog4j.rootLogger={{ env[\"KAFKA_LOG4J_ROOT_LOGLEVEL\"] | default('INFO') }}, stdout\n\nlog4j.appender.stdout=org.apache.log4j.ConsoleAppender\nlog4j.appender.stdout.layout=org.apache.log4j.PatternLayout\nlog4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n\n\n{% set loggers = {\n 'kafka': 'INFO',\n 'kafka.network.RequestChannel$': 'WARN',\n 'kafka.producer.async.DefaultEventHandler': 'DEBUG',\n 'kafka.request.logger': 'WARN',\n 'kafka.controller': 'TRACE',\n 'kafka.log.LogCleaner': 'INFO',\n 'state.change.logger': 'TRACE',\n 'kafka.authorizer.logger': 'WARN'\n } -%}\n\n\n{% if env['KAFKA_LOG4J_LOGGERS'] %}\n{% set loggers = parse_log4j_loggers(env['KAFKA_LOG4J_LOGGERS'], loggers) %}\n{% endif %}\n\n{% for logger,loglevel in loggers.items() %}\nlog4j.logger.{{logger}}={{loglevel}}\n{% endfor %}\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/include/etc/confluent/docker/run",
"content": "#!/usr/bin/env bash\n#\n# Copyright 2016 Confluent Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n. /etc/confluent/docker/bash-config\n\n# Set environment values if they exist as arguments\nif [ $# -ne 0 ]; then\n echo \"===> Overriding env params with args ...\"\n for var in \"$@\"\n do\n export \"$var\"\n done\nfi\n\necho \"===> User\"\nid\n\necho \"===> Configuring ...\"\n/etc/confluent/docker/configure\n\necho \"===> Running preflight checks ... \"\n/etc/confluent/docker/ensure\n\necho \"===> Launching ... \"\nexec /etc/confluent/docker/launch\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/include/etc/confluent/docker/tools-log4j.properties.template",
"content": "\nlog4j.rootLogger={{ env[\"KAFKA_TOOLS_LOG4J_LOGLEVEL\"] | default('WARN') }}, stderr\n\nlog4j.appender.stderr=org.apache.log4j.ConsoleAppender\nlog4j.appender.stderr.layout=org.apache.log4j.PatternLayout\nlog4j.appender.stderr.layout.ConversionPattern=[%d] %p %m (%c)%n\nlog4j.appender.stderr.Target=System.err\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/pom.xml",
"content": "\n\n\n\n 4.0.0\n\n \n io.confluent.kafka-images\n kafka-images-parent\n 7.4.0-0\n \n\n io.confluent.kafka-images\n cp-kafka\n Kafka Docker Image\n\n \n false\n true\n \n\n \n \n junit\n junit\n test\n \n \n\n \n \n \n org.apache.maven.plugins\n maven-jar-plugin\n 2.6\n \n \n none\n \n \n \n \n \n\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/requirements.txt",
"content": "git+https://github.com/confluentinc/confluent-docker-utils@v0.0.32\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/setup.py",
"content": "import setuptools\n\n\nsetuptools.setup(\n name='kafka-tests',\n version='0.0.1',\n author=\"Confluent, Inc.\",\n author_email=\"core-kafka-eng@confluent.io\",\n description='Kafka docker image tests',\n url=\"https://github.com/confluentinc/kafka-images\",\n dependency_links=open(\"requirements.txt\").read().split(\"\\n\"),\n packages=['test'],\n include_package_data=True,\n python_requires='>=2.7',\n setup_requires=['setuptools-git'],\n)\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/cluster-bridged-plain.yml",
"content": "---\nversion: '2'\nnetworks:\n zk:\n driver: bridge\nservices:\n zookeeper-1:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 1\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_CLIENT_PORT: 2181\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: zookeeper-1:2888:3888;zookeeper-2:32888:33888;zookeeper-3:42888:43888\n networks:\n - default\n - zk\n ports:\n - 22181:2181\n - 22888:2888\n - 23888:3888\n labels:\n - io.confluent.docker.testing=true\n\n zookeeper-2:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 2\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_CLIENT_PORT: 2181\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: zookeeper-1:2888:3888;zookeeper-2:32888:33888;zookeeper-3:42888:43888\n networks:\n - default\n - zk\n ports:\n - 32181:2181\n - 32888:2888\n - 33888:3888\n labels:\n - io.confluent.docker.testing=true\n\n zookeeper-3:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 3\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_CLIENT_PORT: 2181\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: zookeeper-1:2888:3888;zookeeper-2:32888:33888;zookeeper-3:42888:43888\n networks:\n - default\n - zk\n ports:\n - 42181:2181\n - 42888:2888\n - 43888:3888\n labels:\n - io.confluent.docker.testing=true\n\n kafka-1:\n image: confluentinc/cp-kafka:latest\n networks:\n - default\n - zk\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:9092\n labels:\n - io.confluent.docker.testing=true\n\n kafka-2:\n image: confluentinc/cp-kafka:latest\n networks:\n - default\n - zk\n environment:\n KAFKA_BROKER_ID: 2\n KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:9092\n labels:\n - io.confluent.docker.testing=true\n\n kafka-3:\n image: confluentinc/cp-kafka:latest\n networks:\n - default\n - zk\n environment:\n KAFKA_BROKER_ID: 3\n KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-3:9092\n labels:\n - io.confluent.docker.testing=true\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/cluster-bridged-sasl.yml",
"content": "---\nversion: '2'\nnetworks:\n zk:\n driver: bridge\nservices:\n zookeeper-1:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 1\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_CLIENT_PORT: 2181\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: zookeeper-1:2888:3888;zookeeper-2:32888:33888;zookeeper-3:42888:43888\n networks:\n - default\n - zk\n ports:\n - 22181:2181\n - 22888:2888\n - 23888:3888\n labels:\n - io.confluent.docker.testing=true\n\n zookeeper-2:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 2\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_CLIENT_PORT: 2181\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: zookeeper-1:2888:3888;zookeeper-2:32888:33888;zookeeper-3:42888:43888\n networks:\n - default\n - zk\n ports:\n - 32181:2181\n - 32888:2888\n - 33888:3888\n labels:\n - io.confluent.docker.testing=true\n\n zookeeper-3:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 3\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_CLIENT_PORT: 2181\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: zookeeper-1:2888:3888;zookeeper-2:32888:33888;zookeeper-3:42888:43888\n networks:\n - default\n - zk\n ports:\n - 42181:2181\n - 42888:2888\n - 43888:3888\n labels:\n - io.confluent.docker.testing=true\n\n kerberos:\n image: confluentinc/cp-kerberos\n networks:\n - default\n - zk\n environment:\n BOOTSTRAP: 0\n volumes:\n - /tmp/kafka-cluster-bridge-test/secrets:/tmp/keytab\n - /dev/urandom:/dev/random\n labels:\n - io.confluent.docker.testing=true\n\n kafka-sasl-ssl-1:\n image: confluentinc/cp-kafka:latest\n hostname: kafka-sasl-ssl-1\n networks:\n - default\n - zk\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/saslssl\n KAFKA_ADVERTISED_LISTENERS: SASL_SSL://kafka-sasl-ssl-1:9094\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker1.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker1_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker1_sslkey_creds\n KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker1.truststore.jks\n KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker1_truststore_creds\n KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SASL_SSL\n KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: GSSAPI\n KAFKA_SASL_ENABLED_MECHANISMS: GSSAPI\n KAFKA_SASL_KERBEROS_SERVICE_NAME: kafka\n KAFKA_LOG4J_ROOT_LOGLEVEL: DEBUG\n ZOOKEEPER_SASL_ENABLED: 'FALSE'\n KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/secrets/bridged_broker1_jaas.conf\n -Djava.security.krb5.conf=/etc/kafka/secrets/bridged_krb.conf -Dsun.net.spi.nameservice.provider.1=sun\n -Dsun.security.krb5.debug=true\n volumes:\n - /tmp/kafka-cluster-bridge-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n\n kafka-sasl-ssl-2:\n image: confluentinc/cp-kafka:latest\n hostname: kafka-sasl-ssl-2\n networks:\n - default\n - zk\n environment:\n KAFKA_BROKER_ID: 2\n KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/saslssl\n KAFKA_ADVERTISED_LISTENERS: SASL_SSL://kafka-sasl-ssl-2:9094\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker2.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker2_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker2_sslkey_creds\n KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker2.truststore.jks\n KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker2_truststore_creds\n KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SASL_SSL\n KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: GSSAPI\n KAFKA_SASL_ENABLED_MECHANISMS: GSSAPI\n KAFKA_SASL_KERBEROS_SERVICE_NAME: kafka\n KAFKA_LOG4J_ROOT_LOGLEVEL: DEBUG\n ZOOKEEPER_SASL_ENABLED: 'FALSE'\n KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/secrets/bridged_broker2_jaas.conf\n -Djava.security.krb5.conf=/etc/kafka/secrets/bridged_krb.conf -Dsun.net.spi.nameservice.provider.1=sun\n -Dsun.security.krb5.debug=true\n volumes:\n - /tmp/kafka-cluster-bridge-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n\n kafka-sasl-ssl-3:\n image: confluentinc/cp-kafka:latest\n hostname: kafka-sasl-ssl-3\n networks:\n - default\n - zk\n environment:\n KAFKA_BROKER_ID: 3\n KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/saslssl\n KAFKA_ADVERTISED_LISTENERS: SASL_SSL://kafka-sasl-ssl-3:9094\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker3.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker3_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker3_sslkey_creds\n KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker3.truststore.jks\n KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker3_truststore_creds\n KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SASL_SSL\n KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: GSSAPI\n KAFKA_SASL_ENABLED_MECHANISMS: GSSAPI\n KAFKA_SASL_KERBEROS_SERVICE_NAME: kafka\n KAFKA_OG4J_ROOT_LOGLEVEL: DEBUG\n ZOOKEEPER_SASL_ENABLED: 'FALSE'\n KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/secrets/bridged_broker3_jaas.conf\n -Djava.security.krb5.conf=/etc/kafka/secrets/bridged_krb.conf -Dsun.net.spi.nameservice.provider.1=sun\n -Dsun.security.krb5.debug=true\n volumes:\n - /tmp/kafka-cluster-bridge-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/cluster-bridged-ssl.yml",
"content": "---\nversion: '2'\nnetworks:\n zk:\n driver: bridge\nservices:\n zookeeper-1:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 1\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_CLIENT_PORT: 2181\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: zookeeper-1:2888:3888;zookeeper-2:32888:33888;zookeeper-3:42888:43888\n networks:\n - default\n - zk\n ports:\n - 22181:2181\n - 22888:2888\n - 23888:3888\n labels:\n - io.confluent.docker.testing=true\n\n zookeeper-2:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 2\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_CLIENT_PORT: 2181\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: zookeeper-1:2888:3888;zookeeper-2:32888:33888;zookeeper-3:42888:43888\n networks:\n - default\n - zk\n ports:\n - 32181:2181\n - 32888:2888\n - 33888:3888\n labels:\n - io.confluent.docker.testing=true\n\n zookeeper-3:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 3\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_CLIENT_PORT: 2181\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: zookeeper-1:2888:3888;zookeeper-2:32888:33888;zookeeper-3:42888:43888\n networks:\n - default\n - zk\n ports:\n - 42181:2181\n - 42888:2888\n - 43888:3888\n labels:\n - io.confluent.docker.testing=true\n\n kafka-ssl-1:\n image: confluentinc/cp-kafka:latest\n networks:\n - default\n - zk\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/ssl\n KAFKA_ADVERTISED_LISTENERS: SSL://kafka-ssl-1:9093\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker1.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker1_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker1_sslkey_creds\n KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker1.truststore.jks\n KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker1_truststore_creds\n KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL\n volumes:\n - /tmp/kafka-cluster-bridge-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n\n kafka-ssl-2:\n image: confluentinc/cp-kafka:latest\n networks:\n - default\n - zk\n environment:\n KAFKA_BROKER_ID: 2\n KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/ssl\n KAFKA_ADVERTISED_LISTENERS: SSL://kafka-ssl-2:9093\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker2.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker2_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker2_sslkey_creds\n KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker2.truststore.jks\n KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker2_truststore_creds\n KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL\n volumes:\n - /tmp/kafka-cluster-bridge-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n\n kafka-ssl-3:\n image: confluentinc/cp-kafka:latest\n networks:\n - default\n - zk\n environment:\n KAFKA_BROKER_ID: 3\n KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/ssl\n KAFKA_ADVERTISED_LISTENERS: SSL://kafka-ssl-3:9093\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker3.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker3_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker3_sslkey_creds\n KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker3.truststore.jks\n KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker3_truststore_creds\n KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL\n volumes:\n - /tmp/kafka-cluster-bridge-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/cluster-host-plain.yml",
"content": "---\nversion: '2'\nservices:\n zookeeper-1:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 1\n ZOOKEEPER_CLIENT_PORT: 22181\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888\n network_mode: host\n labels:\n - io.confluent.docker.testing=true\n\n zookeeper-2:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 2\n ZOOKEEPER_CLIENT_PORT: 32181\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888\n network_mode: host\n labels:\n - io.confluent.docker.testing=true\n\n zookeeper-3:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 3\n ZOOKEEPER_CLIENT_PORT: 42181\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888\n network_mode: host\n labels:\n - io.confluent.docker.testing=true\n\n kafka-1:\n image: confluentinc/cp-kafka:latest\n network_mode: host\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: localhost:22181,localhost:32181,localhost:42181\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:19092\n labels:\n - io.confluent.docker.testing=true\n\n kafka-2:\n image: confluentinc/cp-kafka:latest\n network_mode: host\n environment:\n KAFKA_BROKER_ID: 2\n KAFKA_ZOOKEEPER_CONNECT: localhost:22181,localhost:32181,localhost:42181\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:29092\n labels:\n - io.confluent.docker.testing=true\n\n kafka-3:\n image: confluentinc/cp-kafka:latest\n network_mode: host\n environment:\n KAFKA_BROKER_ID: 3\n KAFKA_ZOOKEEPER_CONNECT: localhost:22181,localhost:32181,localhost:42181\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:39092\n labels:\n - io.confluent.docker.testing=true\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/cluster-host-sasl.yml",
"content": "---\nversion: '2'\nservices:\n zookeeper-sasl-1:\n image: confluentinc/cp-zookeeper:latest\n # This is required because Zookeeper can fail if kerberos is still initializing.\n restart: on-failure:3\n environment:\n ZOOKEEPER_SERVER_ID: 1\n ZOOKEEPER_CLIENT_PORT: 22181\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: sasl.kafka.com:22888:23888;sasl.kafka.com:32888:33888;sasl.kafka.com:42888:43888\n KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/secrets/host_zookeeper_1_jaas.conf\n -Djava.security.krb5.conf=/etc/kafka/secrets/host_krb.conf\n -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider\n -Dsun.security.krb5.debug=true\n volumes:\n - /tmp/kafka-cluster-host-test/secrets:/etc/kafka/secrets\n network_mode: host\n labels:\n - io.confluent.docker.testing=true\n\n zookeeper-sasl-2:\n image: confluentinc/cp-zookeeper:latest\n # This is required because Zookeeper can fail if kerberos is still initializing.\n restart: on-failure:3\n environment:\n ZOOKEEPER_SERVER_ID: 2\n ZOOKEEPER_CLIENT_PORT: 32181\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: sasl.kafka.com:22888:23888;sasl.kafka.com:32888:33888;sasl.kafka.com:42888:43888\n KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/secrets/host_zookeeper_2_jaas.conf\n -Djava.security.krb5.conf=/etc/kafka/secrets/host_krb.conf\n -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider\n -Dsun.security.krb5.debug=true\n volumes:\n - /tmp/kafka-cluster-host-test/secrets:/etc/kafka/secrets\n network_mode: host\n labels:\n - io.confluent.docker.testing=true\n\n zookeeper-sasl-3:\n image: confluentinc/cp-zookeeper:latest\n # This is required because Zookeeper can fail if kerberos is still initializing.\n restart: on-failure:3\n environment:\n ZOOKEEPER_SERVER_ID: 3\n ZOOKEEPER_CLIENT_PORT: 42181\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: sasl.kafka.com:22888:23888;sasl.kafka.com:32888:33888;sasl.kafka.com:42888:43888\n KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/secrets/host_zookeeper_3_jaas.conf\n -Djava.security.krb5.conf=/etc/kafka/secrets/host_krb.conf\n -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider\n -Dsun.security.krb5.debug=true\n volumes:\n - /tmp/kafka-cluster-host-test/secrets:/etc/kafka/secrets\n network_mode: host\n labels:\n - io.confluent.docker.testing=true\n\n kerberos:\n image: confluentinc/cp-kerberos\n network_mode: host\n environment:\n BOOTSTRAP: 0\n volumes:\n - /tmp/kafka-cluster-host-test/secrets:/tmp/keytab\n - /dev/urandom:/dev/random\n labels:\n - io.confluent.docker.testing=true\n\n kafka-sasl-ssl-1:\n image: confluentinc/cp-kafka:latest\n network_mode: host\n # This is required because Kafka can fail if kerberos is still initializing.\n restart: on-failure:3\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: sasl.kafka.com:22181,sasl.kafka.com:32181,sasl.kafka.com:42181/saslssl\n KAFKA_ADVERTISED_LISTENERS: SASL_SSL://sasl.kafka.com:19094\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker1.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker1_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker1_sslkey_creds\n KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker1.truststore.jks\n KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker1_truststore_creds\n KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SASL_SSL\n KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: GSSAPI\n KAFKA_SASL_ENABLED_MECHANISMS: GSSAPI\n KAFKA_SASL_KERBEROS_SERVICE_NAME: kafka\n KAFKA_LOG4J_ROOT_LOGLEVEL: DEBUG\n KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/secrets/host_broker1_jaas.conf\n -Djava.security.krb5.conf=/etc/kafka/secrets/host_krb.conf\n -Dsun.security.krb5.debug=true\n volumes:\n - /tmp/kafka-cluster-host-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n\n kafka-sasl-ssl-2:\n image: confluentinc/cp-kafka:latest\n network_mode: host\n restart: on-failure:3\n environment:\n KAFKA_BROKER_ID: 2\n KAFKA_ZOOKEEPER_CONNECT: sasl.kafka.com:22181,sasl.kafka.com:32181,sasl.kafka.com:42181/saslssl\n KAFKA_ADVERTISED_LISTENERS: SASL_SSL://sasl.kafka.com:29094\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker2.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker2_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker2_sslkey_creds\n KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker2.truststore.jks\n KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker2_truststore_creds\n KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SASL_SSL\n KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: GSSAPI\n KAFKA_SASL_ENABLED_MECHANISMS: GSSAPI\n KAFKA_SASL_KERBEROS_SERVICE_NAME: kafka\n KAFKA_LOG4J_ROOT_LOGLEVEL: DEBUG\n KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/secrets/host_broker2_jaas.conf\n -Djava.security.krb5.conf=/etc/kafka/secrets/host_krb.conf\n -Dsun.security.krb5.debug=true\n volumes:\n - /tmp/kafka-cluster-host-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n\n kafka-sasl-ssl-3:\n image: confluentinc/cp-kafka:latest\n network_mode: host\n restart: on-failure:3\n environment:\n KAFKA_BROKER_ID: 3\n KAFKA_ZOOKEEPER_CONNECT: sasl.kafka.com:22181,sasl.kafka.com:32181,sasl.kafka.com:42181/saslssl\n KAFKA_ADVERTISED_LISTENERS: SASL_SSL://sasl.kafka.com:39094\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker3.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker3_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker3_sslkey_creds\n KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker3.truststore.jks\n KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker3_truststore_creds\n KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SASL_SSL\n KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: GSSAPI\n KAFKA_SASL_ENABLED_MECHANISMS: GSSAPI\n KAFKA_SASL_KERBEROS_SERVICE_NAME: kafka\n KAFKA_LOG4J_ROOT_LOGLEVEL: DEBUG\n KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/secrets/host_broker3_jaas.conf\n -Djava.security.krb5.conf=/etc/kafka/secrets/host_krb.conf\n -Dsun.security.krb5.debug=true\n volumes:\n - /tmp/kafka-cluster-host-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/cluster-host-ssl.yml",
"content": "---\nversion: '2'\nservices:\n zookeeper-1:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 1\n ZOOKEEPER_CLIENT_PORT: 22181\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888\n network_mode: host\n labels:\n - io.confluent.docker.testing=true\n\n zookeeper-2:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 2\n ZOOKEEPER_CLIENT_PORT: 32181\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888\n network_mode: host\n labels:\n - io.confluent.docker.testing=true\n\n zookeeper-3:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 3\n ZOOKEEPER_CLIENT_PORT: 42181\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_INIT_LIMIT: 5\n ZOOKEEPER_SYNC_LIMIT: 2\n ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888\n network_mode: host\n labels:\n - io.confluent.docker.testing=true\n\n kafka-ssl-1:\n image: confluentinc/cp-kafka:latest\n network_mode: host\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: localhost:22181,localhost:32181,localhost:42181/ssl\n KAFKA_ADVERTISED_LISTENERS: SSL://localhost:19093\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker1.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker1_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker1_sslkey_creds\n KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker1.truststore.jks\n KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker1_truststore_creds\n KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL\n volumes:\n - /tmp/kafka-cluster-host-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n\n kafka-ssl-2:\n image: confluentinc/cp-kafka:latest\n network_mode: host\n environment:\n KAFKA_BROKER_ID: 2\n KAFKA_ZOOKEEPER_CONNECT: localhost:22181,localhost:32181,localhost:42181/ssl\n KAFKA_ADVERTISED_LISTENERS: SSL://localhost:29093\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker2.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker2_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker2_sslkey_creds\n KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker2.truststore.jks\n KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker2_truststore_creds\n KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL\n volumes:\n - /tmp/kafka-cluster-host-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n\n kafka-ssl-3:\n image: confluentinc/cp-kafka:latest\n network_mode: host\n environment:\n KAFKA_BROKER_ID: 3\n KAFKA_ZOOKEEPER_CONNECT: localhost:22181,localhost:32181,localhost:42181/ssl\n KAFKA_ADVERTISED_LISTENERS: SSL://localhost:39093\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker3.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker3_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker3_sslkey_creds\n KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker3.truststore.jks\n KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker3_truststore_creds\n KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL\n volumes:\n - /tmp/kafka-cluster-host-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/bridged.consumer.ssl.config",
"content": "group.id=ssl-bridged\nssl.truststore.location=/etc/kafka/secrets/kafka.consumer.truststore.jks\nssl.truststore.password=confluent\n\nssl.keystore.location=/etc/kafka/secrets/kafka.consumer.keystore.jks\nssl.keystore.password=confluent\nssl.key.password=confluent\n\nsecurity.protocol=SSL\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/bridged.consumer.ssl.sasl.config",
"content": "group.id=ssl-sasl-bridged\nssl.truststore.location=/etc/kafka/secrets/kafka.consumer.truststore.jks\nssl.truststore.password=confluent\n\nssl.keystore.location=/etc/kafka/secrets/kafka.consumer.keystore.jks\nssl.keystore.password=confluent\nssl.key.password=confluent\n\nsecurity.protocol=SASL_SSL\nsasl.mechanism=GSSAPI\nsasl.kerberos.service.name=kafka\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/bridged.producer.ssl.config",
"content": "ssl.truststore.location=/etc/kafka/secrets/kafka.producer.truststore.jks\nssl.truststore.password=confluent\n\nssl.keystore.location=/etc/kafka/secrets/kafka.producer.keystore.jks\nssl.keystore.password=confluent\nssl.key.password=confluent\n\nsecurity.protocol=SSL\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/bridged.producer.ssl.sasl.config",
"content": "ssl.truststore.location=/etc/kafka/secrets/kafka.producer.truststore.jks\nssl.truststore.password=confluent\n\nssl.keystore.location=/etc/kafka/secrets/kafka.producer.keystore.jks\nssl.keystore.password=confluent\nssl.key.password=confluent\n\nsecurity.protocol=SASL_SSL\nsasl.mechanism=GSSAPI\nsasl.kerberos.service.name=kafka\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/bridged_broker1_jaas.conf",
"content": "KafkaServer {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/bridged_broker1.keytab\"\n principal=\"kafka/kafka-sasl-ssl-1@TEST.CONFLUENT.IO\";\n};\n\nKafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/bridged_broker1.keytab\"\n principal=\"kafka/kafka-sasl-ssl-1@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/bridged_broker2_jaas.conf",
"content": "KafkaServer {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/bridged_broker2.keytab\"\n principal=\"kafka/kafka-sasl-ssl-2@TEST.CONFLUENT.IO\";\n};\n\nKafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/bridged_broker2.keytab\"\n principal=\"kafka/kafka-sasl-ssl-2@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/bridged_broker3_jaas.conf",
"content": "KafkaServer {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/bridged_broker3.keytab\"\n principal=\"kafka/kafka-sasl-ssl-3@TEST.CONFLUENT.IO\";\n};\n\nKafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/bridged_broker3.keytab\"\n principal=\"kafka/kafka-sasl-ssl-3@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/bridged_consumer_jaas.conf",
"content": "KafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/bridged_consumer.keytab\"\n principal=\"bridged_consumer/kafka-sasl-ssl-consumer@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/bridged_krb.conf",
"content": "[logging]\n default = FILE:/var/log/kerberos/krb5libs.log\n kdc = FILE:/var/log/kerberos/krb5kdc.log\n admin_server = FILE:/var/log/kerberos/kadmind.log\n\n[libdefaults]\n default_realm = TEST.CONFLUENT.IO\n dns_lookup_realm = false\n dns_lookup_kdc = false\n ticket_lifetime = 24h\n renew_lifetime = 7d\n forwardable = true\n # WARNING: We use weaker key types to simplify testing as stronger key types\n # require the enhanced security JCE policy file to be installed. You should\n # NOT run with this configuration in production or any real environment. You\n # have been warned.\n default_tkt_enctypes = des-cbc-md5 des-cbc-crc des3-cbc-sha1\n default_tgs_enctypes = des-cbc-md5 des-cbc-crc des3-cbc-sha1\n permitted_enctypes = des-cbc-md5 des-cbc-crc des3-cbc-sha1\n\n[realms]\n TEST.CONFLUENT.IO = {\n kdc = kerberos\n admin_server = kerberos\n }\n\n[domain_realm]\n .test.confluent.io = TEST.CONFLUENT.IO\n test.confluent.io = TEST.CONFLUENT.IO\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/bridged_producer_jaas.conf",
"content": "KafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/bridged_producer.keytab\"\n principal=\"bridged_producer/kafka-sasl-ssl-producer@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/broker1-ca1-signed.crt",
"content": "-----BEGIN CERTIFICATE-----\nMIIC0jCCAjsCCQC4Ge6Xmxv2ajANBgkqhkiG9w0BAQUFADBjMR4wHAYDVQQDExVj\nYTEudGVzdC5jb25mbHVlbnQuaW8xDTALBgNVBAsTBFRFU1QxEjAQBgNVBAoTCUNP\nTkZMVUVOVDERMA8GA1UEBxMIUGFsb0FsdG8xCzAJBgNVBAYTAlVTMB4XDTE2MDcw\nOTE4MTQyOVoXDTQzMTEyNDE4MTQyOVowdDELMAkGA1UEBhMCVVMxCzAJBgNVBAgT\nAkNhMREwDwYDVQQHEwhQYWxvQWx0bzESMBAGA1UEChMJQ09ORkxVRU5UMQ0wCwYD\nVQQLEwRURVNUMSIwIAYDVQQDExlicm9rZXIxLnRlc3QuY29uZmx1ZW50LmlvMIIB\nIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAoxhAcghALuWHLZtKFjSKoA6z\nEUE5djS4iudQ7hHLV68JReMWJR4fO2OfsZHKNo8uIzF836SgH0ZYepmt0PtRLspi\nkKmSwlJQjBmB9/JSOnUuWX53DxWaIKZUaB/OwdxPgo3qpLXciGwOffip68loo7XG\nbhYStZfDCiDw2w+N7Px93a9xA0ZNgWHFsal8qoMLg3V0xW3BkS+jcjsPud2muXOq\nE1a93P/40ZkgfoGkFHvh+HJITXhVtRDIoJJMJO6UFr7jfqnhvC07nDVCJjCIOl7C\nebid3e2gplBwsyBeL9ulc6EfMa/URaAFvGMkIy0Qkcr9hlPHPcNvnglVvISocQID\nAQABMA0GCSqGSIb3DQEBBQUAA4GBADXwLG913lSI05RqyT0Ph/mtA4NyfPUkOnuJ\nJxJzHWzp+G68QjmWMGycN6fqN1MosjZtu9/p4Z5Rjx2ywJCOO4wcOteLuIvJkfHm\n4gB+NvzajUZ1YIg/LD09TPSYXmPX5juR/zxMaChcZigGTpADnpoLfsyydUNSLw3L\nboKeqh+v\n-----END CERTIFICATE-----\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/broker1_keystore_creds",
"content": "confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/broker1_sslkey_creds",
"content": "confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/broker1_truststore_creds",
"content": "confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/broker2-ca1-signed.crt",
"content": "-----BEGIN CERTIFICATE-----\nMIIC0jCCAjsCCQC4Ge6Xmxv2azANBgkqhkiG9w0BAQUFADBjMR4wHAYDVQQDExVj\nYTEudGVzdC5jb25mbHVlbnQuaW8xDTALBgNVBAsTBFRFU1QxEjAQBgNVBAoTCUNP\nTkZMVUVOVDERMA8GA1UEBxMIUGFsb0FsdG8xCzAJBgNVBAYTAlVTMB4XDTE2MDcw\nOTE4MTQzM1oXDTQzMTEyNDE4MTQzM1owdDELMAkGA1UEBhMCVVMxCzAJBgNVBAgT\nAkNhMREwDwYDVQQHEwhQYWxvQWx0bzESMBAGA1UEChMJQ09ORkxVRU5UMQ0wCwYD\nVQQLEwRURVNUMSIwIAYDVQQDExlicm9rZXIyLnRlc3QuY29uZmx1ZW50LmlvMIIB\nIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAud6NF7S4nl4USma/Dvxq0Ftx\n1OiuGZVKw5qo3KL+v87GP2Uyoqqu7YsB/dFpfl8j49My3Bo4Jy5g60oGgHEYh0lj\n0vb0NqNKUSu4/7jNHuN4hGnplPMTtshlrU0VaDCKM6M1IHJYsGzcpsEoVmjA5NaT\nzq9WKKXzdbM/5n0NHvvnESyR9Ug1RKuEPAGKxqc0AwMHiqWCLb8jHtbjkzf7PuEy\nMIAzeb/BcJygqBYvKHqQKtumWQy0YdT5vDl4M7Aywv2p188s5vBEgzCmjCQRjPph\n1lLBWka1OGoebmPt5DqWNvtHzXA1Bit2aOA3BxnXN50Jq0MtEq46NGCQ4Up8XQID\nAQABMA0GCSqGSIb3DQEBBQUAA4GBAIVf4TckPp7iGjMIx1wJjPv9RSPynF/hljgS\n7indUs9vjijLMuMR4E66a/JQiKqZxGUefBvtYX0oGKfRZDO5DgwHx50Kv9Yx6Ux6\nVqTq+CFpWJLSsdlNHHbjVYCVjCEFZAUX/Y4ULVCNihYNaLIB+NWNj5jEuemqAnUl\neyM5UwLQ\n-----END CERTIFICATE-----\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/broker2_keystore_creds",
"content": "confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/broker2_sslkey_creds",
"content": "confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/broker2_truststore_creds",
"content": "confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/broker3-ca1-signed.crt",
"content": "-----BEGIN CERTIFICATE-----\nMIIC0jCCAjsCCQC4Ge6Xmxv2bDANBgkqhkiG9w0BAQUFADBjMR4wHAYDVQQDExVj\nYTEudGVzdC5jb25mbHVlbnQuaW8xDTALBgNVBAsTBFRFU1QxEjAQBgNVBAoTCUNP\nTkZMVUVOVDERMA8GA1UEBxMIUGFsb0FsdG8xCzAJBgNVBAYTAlVTMB4XDTE2MDcw\nOTE4MTQzNloXDTQzMTEyNDE4MTQzNlowdDELMAkGA1UEBhMCVVMxCzAJBgNVBAgT\nAkNhMREwDwYDVQQHEwhQYWxvQWx0bzESMBAGA1UEChMJQ09ORkxVRU5UMQ0wCwYD\nVQQLEwRURVNUMSIwIAYDVQQDExlicm9rZXIzLnRlc3QuY29uZmx1ZW50LmlvMIIB\nIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAhFw0H4/NksLIyFu+8YnrKr9J\nD1iupm5mpavXFiB3UrCUYjBPc+bkXRExFtAnhwB1q7R/E+zJtzlmLl5FojWFSaiv\nt2bjugPHs6bZ1D3p0UqouoQf7AgQQNSB+wZoZCp3yiKxI+s1U3NKT6MRbAN5eaUx\n1JNo1fV2zVIIvsKFe1Zldt5uSML9OBtcixaJCsPkyGenV1kajkoiHC8UHgzu5obr\n9QSOdkRWTgq5LX+gyWhAC4hF+ApA5QGQYT8m5paj5c2YHhpZcTFepRnZSh3fq7Xq\nhXPqzQyX6v9Kxii9QaVwY2zwgON09OJ5KF9UK4FPQfZmd4dJEVV4CybhHTP3kQID\nAQABMA0GCSqGSIb3DQEBBQUAA4GBANstHkSvQjumHlwQSAaQ4pA6YION0GcY+Lzl\nvUIE2DFRwzsV87wFa2sc46XOSpjhUxaYEqtyzHYCaPaZ/n2t07857AqNXJjeJZhW\nL/l17cAFdPToP63cpMBQF9deQyhHQTMEMhPKBYg9ym9B3wh2emGSxriD1nhU0cbK\nKlEkqnUP\n-----END CERTIFICATE-----\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/broker3_keystore_creds",
"content": "confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/broker3_sslkey_creds",
"content": "confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/broker3_truststore_creds",
"content": "confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/client-plain.config",
"content": ""
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/config_krb.conf",
"content": "[logging]\n default = FILE:/var/log/kerberos/krb5libs.log\n kdc = FILE:/var/log/kerberos/krb5kdc.log\n admin_server = FILE:/var/log/kerberos/kadmind.log\n\n[libdefaults]\n default_realm = TEST.CONFLUENT.IO\n dns_lookup_realm = false\n dns_lookup_kdc = false\n ticket_lifetime = 24h\n renew_lifetime = 7d\n forwardable = true\n\n[realms]\n TEST.CONFLUENT.IO = {\n kdc = kerberos\n admin_server = confluent\n }\n\n[domain_realm]\n .test.confluent.io = TEST.CONFLUENT.IO\n test.confluent.io = TEST.CONFLUENT.IO\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/config_server1_jaas.conf",
"content": "KafkaServer {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/broker1.keytab\"\n principal=\"kafka/sasl-ssl-config@TEST.CONFLUENT.IO\";\n};\nKafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/broker1.keytab\"\n principal=\"kafka/sasl-ssl-config@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/consumer-ca1-signed.crt",
"content": "-----BEGIN CERTIFICATE-----\nMIIC0zCCAjwCCQC4Ge6Xmxv2bjANBgkqhkiG9w0BAQUFADBjMR4wHAYDVQQDExVj\nYTEudGVzdC5jb25mbHVlbnQuaW8xDTALBgNVBAsTBFRFU1QxEjAQBgNVBAoTCUNP\nTkZMVUVOVDERMA8GA1UEBxMIUGFsb0FsdG8xCzAJBgNVBAYTAlVTMB4XDTE2MDcw\nOTE4MTQ0MloXDTQzMTEyNDE4MTQ0MlowdTELMAkGA1UEBhMCVVMxCzAJBgNVBAgT\nAkNhMREwDwYDVQQHEwhQYWxvQWx0bzESMBAGA1UEChMJQ09ORkxVRU5UMQ0wCwYD\nVQQLEwRURVNUMSMwIQYDVQQDExpjb25zdW1lci50ZXN0LmNvbmZsdWVudC5pbzCC\nASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJMlGunTtQd2dTY+EPTXMLvO\n+QSznU/JdLM0liqBGInJ2J1yC76avKjBhXqbJRA+cYYq7EvBFuaeFAeRPfTXLJYv\n67cmVN2QSAmd5jGiyOkP2w3q9WYyhczIQLE87NcC0/E3UE9SY25sUsZneJifhJLC\nJpEaQS+JSP8yWMwyGm67ccTIHanvGoha0s2aP97BhTqxAarBzSjW/IDO4r5yCwPJ\ndAbiI00rnt7zgeLwjzJBrvpzYbp4IhAQy8dnPeZ3PbL4qz+tyJECONn7lYIFFHMX\nSVw9GZe4KGnCyhnY/t4aKXRSTk8mBEjIybhfXV/mEE94z+KOAwUWV+uXWLQRUDUC\nAwEAATANBgkqhkiG9w0BAQUFAAOBgQBi9i8ynwSosbyP5FvX1wAHs3QYrD/izFxj\nd8W8STg7/TJbb8jJwl5ievUASiH0rvHrorEHs3Vyijc5W6nLoAL+KLjUYlNyd2b+\n78YYxgUFMFAzvrXJ1oFOWTkDb66LGWCj75z6hKPX2U33PY5+A7YSNMnjYlqQBSgN\nsorDym1cfQ==\n-----END CERTIFICATE-----\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/consumer_keystore_creds",
"content": "confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/consumer_sslkey_creds",
"content": "confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/consumer_truststore_creds",
"content": "confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/create-certs.sh",
"content": "#!/bin/bash\n\nset -o nounset \\\n -o errexit \\\n -o verbose \\\n -o xtrace\n\n# Generate CA key\nopenssl req -new -x509 -keyout snakeoil-ca-1.key -out snakeoil-ca-1.crt -days 365 -subj '/CN=ca1.test.confluent.io/OU=TEST/O=CONFLUENT/L=PaloAlto/S=Ca/C=US' -passin pass:confluent -passout pass:confluent\n# openssl req -new -x509 -keyout snakeoil-ca-2.key -out snakeoil-ca-2.crt -days 365 -subj '/CN=ca2.test.confluent.io/OU=TEST/O=CONFLUENT/L=PaloAlto/S=Ca/C=US' -passin pass:confluent -passout pass:confluent\n\n# Kafkacat\nopenssl genrsa -des3 -passout \"pass:confluent\" -out kafkacat.client.key 1024\nopenssl req -passin \"pass:confluent\" -passout \"pass:confluent\" -key kafkacat.client.key -new -out kafkacat.client.req -subj '/CN=kafkacat.test.confluent.io/OU=TEST/O=CONFLUENT/L=PaloAlto/S=Ca/C=US'\nopenssl x509 -req -CA snakeoil-ca-1.crt -CAkey snakeoil-ca-1.key -in kafkacat.client.req -out kafkacat-ca1-signed.pem -days 9999 -CAcreateserial -passin \"pass:confluent\"\n\n\n\nfor i in broker1 broker2 broker3 producer consumer\ndo\n\techo $i\n\t# Create keystores\n\tkeytool -genkey -noprompt \\\n\t\t\t\t -alias $i \\\n\t\t\t\t -dname \"CN=$i.test.confluent.io, OU=TEST, O=CONFLUENT, L=PaloAlto, S=Ca, C=US\" \\\n\t\t\t\t -keystore kafka.$i.keystore.jks \\\n\t\t\t\t -keyalg RSA \\\n\t\t\t\t -storepass confluent \\\n\t\t\t\t -keypass confluent\n\n\t# Create CSR, sign the key and import back into keystore\n\tkeytool -keystore kafka.$i.keystore.jks -alias $i -certreq -file $i.csr -storepass confluent -keypass confluent\n\n\topenssl x509 -req -CA snakeoil-ca-1.crt -CAkey snakeoil-ca-1.key -in $i.csr -out $i-ca1-signed.crt -days 9999 -CAcreateserial -passin pass:confluent\n\n\tkeytool -keystore kafka.$i.keystore.jks -alias CARoot -import -file snakeoil-ca-1.crt -storepass confluent -keypass confluent\n\n\tkeytool -keystore kafka.$i.keystore.jks -alias $i -import -file $i-ca1-signed.crt -storepass confluent -keypass confluent\n\n\t# Create truststore and import the CA cert.\n\tkeytool -keystore kafka.$i.truststore.jks -alias CARoot -import -file snakeoil-ca-1.crt -storepass confluent -keypass confluent\n\n echo \"confluent\" > ${i}_sslkey_creds\n echo \"confluent\" > ${i}_keystore_creds\n echo \"confluent\" > ${i}_truststore_creds\ndone\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/host.consumer.ssl.config",
"content": "group.id=ssl-host\nssl.truststore.location=/etc/kafka/secrets/kafka.consumer.truststore.jks\nssl.truststore.password=confluent\n\nssl.keystore.location=/etc/kafka/secrets/kafka.consumer.keystore.jks\nssl.keystore.password=confluent\nssl.key.password=confluent\n\nsecurity.protocol=SSL\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/host.consumer.ssl.sasl.config",
"content": "group.id=ssl-sasl-host\nssl.truststore.location=/etc/kafka/secrets/kafka.consumer.truststore.jks\nssl.truststore.password=confluent\n\nssl.keystore.location=/etc/kafka/secrets/kafka.consumer.keystore.jks\nssl.keystore.password=confluent\nssl.key.password=confluent\n\nsecurity.protocol=SASL_SSL\nsasl.mechanism=GSSAPI\nsasl.kerberos.service.name=kafka\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/host.producer.ssl.config",
"content": "ssl.truststore.location=/etc/kafka/secrets/kafka.producer.truststore.jks\nssl.truststore.password=confluent\n\nssl.keystore.location=/etc/kafka/secrets/kafka.producer.keystore.jks\nssl.keystore.password=confluent\nssl.key.password=confluent\n\nsecurity.protocol=SSL\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/host.producer.ssl.sasl.config",
"content": "ssl.truststore.location=/etc/kafka/secrets/kafka.producer.truststore.jks\nssl.truststore.password=confluent\n\nssl.keystore.location=/etc/kafka/secrets/kafka.producer.keystore.jks\nssl.keystore.password=confluent\nssl.key.password=confluent\n\nsecurity.protocol=SASL_SSL\nsasl.mechanism=GSSAPI\nsasl.kerberos.service.name=kafka\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/host_broker1_jaas.conf",
"content": "KafkaServer {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/host_broker1.keytab\"\n principal=\"kafka/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\nKafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/host_broker1.keytab\"\n principal=\"kafka/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\n\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/zkclient-host-1.keytab\"\n principal=\"zkclient/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/host_broker2_jaas.conf",
"content": "KafkaServer {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/host_broker2.keytab\"\n principal=\"kafka/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\nKafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/host_broker2.keytab\"\n principal=\"kafka/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\n\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/zkclient-host-2.keytab\"\n principal=\"zkclient/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/host_broker3_jaas.conf",
"content": "KafkaServer {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/host_broker3.keytab\"\n principal=\"kafka/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\nKafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/host_broker3.keytab\"\n principal=\"kafka/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\n\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/zkclient-host-3.keytab\"\n principal=\"zkclient/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/host_consumer_jaas.conf",
"content": "KafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/host_consumer.keytab\"\n principal=\"host_consumer/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/host_krb.conf",
"content": "[logging]\n default = FILE:/var/log/kerberos/krb5libs.log\n kdc = FILE:/var/log/kerberos/krb5kdc.log\n admin_server = FILE:/var/log/kerberos/kadmind.log\n\n[libdefaults]\n default_realm = TEST.CONFLUENT.IO\n dns_lookup_realm = false\n dns_lookup_kdc = false\n ticket_lifetime = 24h\n renew_lifetime = 7d\n forwardable = true\n\n[realms]\n TEST.CONFLUENT.IO = {\n kdc = localhost\n admin_server = localhost\n }\n\n[domain_realm]\n .TEST.CONFLUENT.IO = TEST.CONFLUENT.IO\n TEST.CONFLUENT.IO = TEST.CONFLUENT.IO\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/host_producer_jaas.conf",
"content": "KafkaClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/host_producer.keytab\"\n principal=\"host_producer/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/host_zookeeper_1_jaas.conf",
"content": "Server {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/zookeeper-host-1.keytab\"\n principal=\"zookeeper/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/zkclient-host-1.keytab\"\n principal=\"zkclient/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/host_zookeeper_2_jaas.conf",
"content": "Server {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/zookeeper-host-1.keytab\"\n principal=\"zookeeper/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/zkclient-host-1.keytab\"\n principal=\"zkclient/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/host_zookeeper_3_jaas.conf",
"content": "Server {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/zookeeper-host-1.keytab\"\n principal=\"zookeeper/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\nClient {\n com.sun.security.auth.module.Krb5LoginModule required\n useKeyTab=true\n storeKey=true\n keyTab=\"/etc/kafka/secrets/zkclient-host-1.keytab\"\n principal=\"zkclient/sasl.kafka.com@TEST.CONFLUENT.IO\";\n};\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/kafkacat-ca1-signed.pem",
"content": "-----BEGIN CERTIFICATE-----\nMIICQjCCAasCCQC4Ge6Xmxv2aTANBgkqhkiG9w0BAQUFADBjMR4wHAYDVQQDExVj\nYTEudGVzdC5jb25mbHVlbnQuaW8xDTALBgNVBAsTBFRFU1QxEjAQBgNVBAoTCUNP\nTkZMVUVOVDERMA8GA1UEBxMIUGFsb0FsdG8xCzAJBgNVBAYTAlVTMB4XDTE2MDcw\nOTE4MTQyOFoXDTQzMTEyNDE4MTQyOFowaDEjMCEGA1UEAxMaa2Fma2FjYXQudGVz\ndC5jb25mbHVlbnQuaW8xDTALBgNVBAsTBFRFU1QxEjAQBgNVBAoTCUNPTkZMVUVO\nVDERMA8GA1UEBxMIUGFsb0FsdG8xCzAJBgNVBAYTAlVTMIGfMA0GCSqGSIb3DQEB\nAQUAA4GNADCBiQKBgQDGtDOf/EYZY08D82ehsAITjLprXDMGnfuiXxdsiZyqCIxc\nJPM6gKtxzU8DnkWTY5xEnWxjIwDjQGXwBCnXaNBq7kgBL3P13rtnX34ZQar49NX3\n6RR8IUTM5HxDzxOkmg3aZ2dkKWZU5B1VRTZzWA7mxQEZMPjV8DrhHUa3XdWH7wID\nAQABMA0GCSqGSIb3DQEBBQUAA4GBAJHd35NpxhDY43LtmHMqGdObBaiUBuB7jai4\nQRzdq7J+bafQ28sIjXo03lV7YRMd9r0gPBXhWymHH838xJh7TnbpHHyJ/CBjVnqG\nMc+cDTMudNWXOrayYeN1WkF/ufP+gJfRl084Lg8BKFaKntRIW/kG/1CniJYRs/JD\namRxX3iB\n-----END CERTIFICATE-----\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/kafkacat.client.key",
"content": "-----BEGIN RSA PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: DES-EDE3-CBC,F8EEC4C2219A7603\n\nJh4EwEWHP6DwOImmfwjgUODepVQ9RoBD5PZLngTuCMGI8xrWbINGi1ejd3H+61Tx\nbwSirRCwKHqr0JVqvaA8VxZEeIleQ+ZqRASEUhnvGU2gTJ7sEF6KD+4i5n4pKBOQ\n0/F0rIHLB/slongr1JlTP7F22XC3DXkHsmsjtTvkqAPXh4tgM5SGwlE8sYKqL4Py\nmoSQBN2+yGsjP96Mg+mskzvSTucKrCJ9cXnTbCQfimgxk9kjNzcrVZN51r5fiQ+Z\n0x0muyQh/G9gvDKP4xq6PAUopUNlzPutVm0rcY7SIE3Q0WK65AfMvbLYPulnAJua\nJJVYGjF0y/Zze4p7hfJUqzfto5+bwfD/AgwUOzCIpPcARKb7LsYoZNJqqkcuXc00\nXQ8evsA/DF9DWXmWNpE6LOqAe5k38c+NYMC7gIYltoFwPr4tq71jkpJfZ+QetzOf\nTuxuYjcXf8FHx12JgOybeglFjK85aZ2nDgI+F5yUx7jvp/Qkpo4tJ/uXhr2D4m9U\nQ+bcas1PJDv+aYRa9pV09hkHTKb0wlnk3r1Fs0lNjyTKhBqhwrhDwAuaw2uj2tgA\nulX3zHp1vNzTecrDdsUNn99xDLls40Uqh3Wlsg56ck8i18DLRre4fyg6Rk33n369\nhobAb1fujIvBpdtzGbeqvo6YGfqb77JVLrr/f3wt62A0ocoYpm7YbD0RFYKC14Tz\nogLii/59KtIaKIeCc/eHs+WiXoyoLqVjMk4/+TXUrKBsuOwxVeOk1FUlJuJpmo5o\n3b7ehgVXprn0xO8lC9xjMIWhbRUl6scE6LyYiUyX12XBAif2pMhV9w==\n-----END RSA PRIVATE KEY-----\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/krb_server.conf",
"content": "[logging]\n default = FILE:/var/log/kerberos/krb5libs.log\n kdc = FILE:/var/log/kerberos/krb5kdc.log\n admin_server = FILE:/var/log/kerberos/kadmind.log\n\n[libdefaults]\n default_realm = TEST.CONFLUENT.IO\n dns_lookup_realm = false\n dns_lookup_kdc = false\n ticket_lifetime = 24h\n renew_lifetime = 7d\n forwardable = true\n udp_preference_limit = 1000000\n # WARNING: We use weaker key types to simplify testing as stronger key types\n # require the enhanced security JCE policy file to be installed. You should\n # NOT run with this configuration in production or any real environment. You\n # have been warned.\n default_tkt_enctypes = des-cbc-md5 des-cbc-crc des3-cbc-sha1\n default_tgs_enctypes = des-cbc-md5 des-cbc-crc des3-cbc-sha1\n permitted_enctypes = des-cbc-md5 des-cbc-crc des3-cbc-sha1\n\n[realms]\n TEST.CONFLUENT.IO = {\n kdc = kerberos\n admin_server = kerberos\n }\n\n[domain_realm]\n .TEST.CONFLUENT.IO = TEST.CONFLUENT.IO\n TEST.CONFLUENT.IO = TEST.CONFLUENT.IO\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/producer-ca1-signed.crt",
"content": "-----BEGIN CERTIFICATE-----\nMIIC0zCCAjwCCQC4Ge6Xmxv2bTANBgkqhkiG9w0BAQUFADBjMR4wHAYDVQQDExVj\nYTEudGVzdC5jb25mbHVlbnQuaW8xDTALBgNVBAsTBFRFU1QxEjAQBgNVBAoTCUNP\nTkZMVUVOVDERMA8GA1UEBxMIUGFsb0FsdG8xCzAJBgNVBAYTAlVTMB4XDTE2MDcw\nOTE4MTQzOVoXDTQzMTEyNDE4MTQzOVowdTELMAkGA1UEBhMCVVMxCzAJBgNVBAgT\nAkNhMREwDwYDVQQHEwhQYWxvQWx0bzESMBAGA1UEChMJQ09ORkxVRU5UMQ0wCwYD\nVQQLEwRURVNUMSMwIQYDVQQDExpwcm9kdWNlci50ZXN0LmNvbmZsdWVudC5pbzCC\nASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK5M+oKxIzpuCLM3/O/RFTjn\nmkdKAvygCkKDrzLvQHaVUQhWtObUOyzxObk+mVj9SS8K1HpGwX88USdQqHuGBsrZ\n5OHUU9yS5TJ3J+wNR8Wf/ki84Z/tM8NCKv9MxCnSoy6s9Wk4Lk8S1lvAp9sPQhaX\nY9d55z+j3LNKsa8YJqX6XHJc3XtyFMshY2LsFS/s3YPGYl54tVVvGQJ7qxf7cVhI\n/ISH1LVfkhA4XO6KSGf/mBg2XTkcGDO5kHVAnmIfGeZ45B17HAy4UI2LMM0Q7xKm\nmgFDMWpdV47JeM2bfP3wNMSREfkepjyZk5PTvevYZJpQJ7/U841RPIkCgE6CCNkC\nAwEAATANBgkqhkiG9w0BAQUFAAOBgQAMVY7TqXsMXnoVb1aWwmNruOKfAlubS/sQ\n4tfxyY1SMfhBYCRR+ZxlGrXY0GmKfzRUjjaH+8rwHn6WRpI3Qk7IHIU5LO+3jrKh\n3DNraRokKBFz35TBmkEJY7Xc5KOzRA3g5739TvDXwKPNtOsI41GpbOl3HSBjnEGG\n01c4XSxmeQ==\n-----END CERTIFICATE-----\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/producer-ssl.config",
"content": "security.protocol=SSL\nssl.truststore.location=/Users/sumit/code/confluent/cp-docker/security/kafka.producer.truststore.jks\nssl.truststore.password=confluent\n\nssl.keystore.location=/Users/sumit/code/confluent/cp-docker/security/kafka.producer.keystore.jks\nssl.keystore.password=confluent\nssl.key.password=confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/producer_keystore_creds",
"content": "confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/producer_sslkey_creds",
"content": "confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/producer_truststore_creds",
"content": "confluent\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/snakeoil-ca-1.crt",
"content": "-----BEGIN CERTIFICATE-----\nMIIDDTCCAnagAwIBAgIJAPgq7cn3Z8iiMA0GCSqGSIb3DQEBBQUAMGMxHjAcBgNV\nBAMTFWNhMS50ZXN0LmNvbmZsdWVudC5pbzENMAsGA1UECxMEVEVTVDESMBAGA1UE\nChMJQ09ORkxVRU5UMREwDwYDVQQHEwhQYWxvQWx0bzELMAkGA1UEBhMCVVMwHhcN\nMTYwNzA5MTgxNDI4WhcNMTcwNzA5MTgxNDI4WjBjMR4wHAYDVQQDExVjYTEudGVz\ndC5jb25mbHVlbnQuaW8xDTALBgNVBAsTBFRFU1QxEjAQBgNVBAoTCUNPTkZMVUVO\nVDERMA8GA1UEBxMIUGFsb0FsdG8xCzAJBgNVBAYTAlVTMIGfMA0GCSqGSIb3DQEB\nAQUAA4GNADCBiQKBgQDt72SR2FPC1HWqghQO8DNlxjPnqgW6RJhDLiA8+iLGPVYc\nMoBtsxMtMoWVx7WUciOWO2Az/v92J7QPhO8KkdRIrv4yTRTd/sYPA5Ky4P19Rc3l\nZr+iWB73EBkliVzEkXkC5mS9Qsx83bl32+d0fMk/GccKlgtJ5Ramf0RNB9a5EwID\nAQABo4HIMIHFMB0GA1UdDgQWBBR4t05HUOhx4JSsfEF5l6PJ8/pzzzCBlQYDVR0j\nBIGNMIGKgBR4t05HUOhx4JSsfEF5l6PJ8/pzz6FnpGUwYzEeMBwGA1UEAxMVY2Ex\nLnRlc3QuY29uZmx1ZW50LmlvMQ0wCwYDVQQLEwRURVNUMRIwEAYDVQQKEwlDT05G\nTFVFTlQxETAPBgNVBAcTCFBhbG9BbHRvMQswCQYDVQQGEwJVU4IJAPgq7cn3Z8ii\nMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADgYEADLxsXBPIddsnW7nLe3Zb\npvVLSf3cwF6SUewgBvxmrcRbuVynsODJc6P0UU8+z6JyjqcYqBoyuupVksBPa1aJ\nwtB/5YCRmXnz1Af2P2NrlpSs6R4uJuCd47OuGhgoA4TPTDgDt3j9zDncCh2e6S2A\nr8koy4uaoj7cNQMV6OZxgjE=\n-----END CERTIFICATE-----\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/secrets/snakeoil-ca-1.key",
"content": "-----BEGIN RSA PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: DES-EDE3-CBC,1A482A57BA230AD3\n\nB5XZ6aSWAYrUgxLtLB+WlLFRN1EpZohUlKUr5kmEDsr2gaa2ELMuxda6ie84uNoQ\npb5PI0TKhdSf2YY0XBHHxPlZy/1LpRTi0typVE5JSsOOc8EqHE/hTas9C6+/Ipw1\nbEybyBdxdvFyY5M7CIhKeEV2FXZnq+enIUi1iiMMmDRMw22VJRXcI+e+MUALUsD/\nhcJk+5njYao6Cf8l+zTGl3Yxq9/mPqgQwfYxdTwkRk9P73TL7X8YC+BH1N1efJqS\nP6xC0viH2xgrg3j6X0zjGGPRwLYONxtzDrCxJgZOoPKdPGLPtjkxN2JHH40kv0QC\n91gdoDnd2n2L4de1hT8T54vwmh8UwGrYARykt+lIMUjZl/Nd715e6uAXQZYyxM4g\ng2gHDXwp5EKcLsQdJI7CejEWgUnHQUOPXc2JQHXknmHTclgK2OBjgrFKLAQQIpjN\nhbcf4MTOhhPqE9QztnNH9i37rqv+SWyhiSKWODpuruaT9+gXnUAZx7bViNKfC0zB\nyQRxozCEr99CaOvZwClizDsz9kCMWbCMw5g9ISyIHm3oMf+sPwZQVT7i+z9+lGDO\njEp9Aoj+n6ywqVfVZAjE3JzvyX84jtX8K7X608JFAtG05wf0Fmb9QwrtL0gOL7C2\nUuDcKYin2tlOX5JQ2CxWlElA4sCNMufvsYL492CYo+gZ0FxVptyu/r3swCujqKqd\nIyYIgHiNRbgWeZ2QEom9GFWxVnWaUW7807tk+p++3tnYyhQY/lA35cmv2RK6jYR8\nQXv9Vc4159fLEPwxBgHC2njgXU2USN+XvNI3n98KUtmNRF0S7a+npQ==\n-----END RSA PRIVATE KEY-----\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/standalone-config.yml",
"content": "---\nversion: '2'\nservices:\n zookeeper:\n image: confluentinc/cp-zookeeper:latest\n environment:\n ZOOKEEPER_SERVER_ID: 1\n ZOOKEEPER_TICK_TIME: 2000\n ZOOKEEPER_CLIENT_PORT: 2181\n labels:\n - io.confluent.docker.testing=true\n\n failing-config:\n image: confluentinc/cp-kafka:latest\n labels:\n - io.confluent.docker.testing=true\n\n failing-config-zk-connect:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n labels:\n - io.confluent.docker.testing=true\n\n failing-config-adv-listeners:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: something\n labels:\n - io.confluent.docker.testing=true\n\n failing-config-adv-port:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://foo:9092\n KAFKA_ADVERTISED_PORT: kafka\n labels:\n - io.confluent.docker.testing=true\n\n failing-config-adv-hostname:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://foo:9092\n KAFKA_ADVERTISED_HOST: kafka\n labels:\n - io.confluent.docker.testing=true\n\n failing-config-port:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://foo:9092\n KAFKA_PORT: kafka\n labels:\n - io.confluent.docker.testing=true\n\n failing-config-ssl-keystore:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_ADVERTISED_LISTENERS: SSL://foo:9092\n labels:\n - io.confluent.docker.testing=true\n\n failing-config-ssl-key-password:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker1.keystore.jks\n KAFKA_ADVERTISED_LISTENERS: SSL://foo:9092\n volumes:\n - /tmp/kafka-config-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n\n failing-config-ssl-keystore-password:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker1.keystore.jks\n KAFKA_SSL_KEY_CREDENTIALS: broker1_sslkey_creds\n KAFKA_ADVERTISED_LISTENERS: SSL://foo:9092\n volumes:\n - /tmp/kafka-config-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n\n failing-config-ssl-truststore:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_ADVERTISED_LISTENERS: SSL://foo:9092\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker1.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker1_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker1_sslkey_creds\n volumes:\n - /tmp/kafka-config-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n\n failing-config-sasl-jaas:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://foo:9092\n labels:\n - io.confluent.docker.testing=true\n\n failing-config-sasl-missing-prop:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://foo:9092\n KAFKA_OPTS: blah\n labels:\n - io.confluent.docker.testing=true\n\n failing-config-ssl-truststore-password:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_ADVERTISED_LISTENERS: SSL://foo:9092\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker1.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker1_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker1_sslkey_creds\n KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker1.truststore.jks\n volumes:\n - /tmp/kafka-config-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n\n failing-config-host:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://foo:9092\n KAFKA_HOST: kafka\n labels:\n - io.confluent.docker.testing=true\n\n default-config:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181/defaultconfig\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://default-config:9092\n labels:\n - io.confluent.docker.testing=true\n\n full-config:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181/fullconfig\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://full-config:9092\n KAFKA_LOG4J_LOGGERS: kafka.controller=WARN,kafka.foo.bar=DEBUG\n KAFKA_LOG4J_ROOT_LOGLEVEL: WARN\n KAFKA_TOOLS_LOG4J_LOGLEVEL: ERROR\n labels:\n - io.confluent.docker.testing=true\n\n external-volumes:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181/externalvolumes\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://external-volumes:9092\n volumes:\n - /tmp/kafka-config-test/data:/var/lib/kafka/data\n labels:\n - io.confluent.docker.testing=true\n\n random-user:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181/randomuser\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://random-user:9092\n user: '12345'\n labels:\n - io.confluent.docker.testing=true\n\n kitchen-sink:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181/kitchensink\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kitchen-sink:9092\n CONFLUENT_SUPPORT_METRICS_ENABLE: 'false'\n CONFLUENT_SUPPORT_CUSTOMER_ID: c0\n volumes:\n - /tmp/kafka-config-kitchen-sink-test/data:/var/lib/kafka/data\n user: '12345'\n labels:\n - io.confluent.docker.testing=true\n\n ssl-config:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181/sslconfig\n KAFKA_ADVERTISED_LISTENERS: SSL://ssl-config:9092\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker1.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker1_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker1_sslkey_creds\n KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker1.truststore.jks\n KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker1_truststore_creds\n KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL\n volumes:\n - /tmp/kafka-config-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n\n kerberos:\n image: confluentinc/cp-kerberos\n environment:\n BOOTSTRAP: 0\n volumes:\n - /tmp/kafka-config-test/secrets:/tmp/keytab\n - /dev/urandom:/dev/random\n labels:\n - io.confluent.docker.testing=true\n\n sasl-ssl-config:\n image: confluentinc/cp-kafka:latest\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181/sslsaslconfig\n KAFKA_ADVERTISED_LISTENERS: SSL://sasl-ssl-config:9092,SASL_SSL://sasl-ssl-config:9094\n KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker1.keystore.jks\n KAFKA_SSL_KEYSTORE_CREDENTIALS: broker1_keystore_creds\n KAFKA_SSL_KEY_CREDENTIALS: broker1_sslkey_creds\n KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker1.truststore.jks\n KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker1_truststore_creds\n KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SASL_SSL\n KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: GSSAPI\n KAFKA_SASL_ENABLED_MECHANISMS: GSSAPI\n KAFKA_SASL_KERBEROS_SERVICE_NAME: kafka\n ZOOKEEPER_SASL_ENABLED: \"FALSE\"\n KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/secrets/config_server1_jaas.conf\n -Djava.security.krb5.conf=/etc/kafka/secrets/config_krb.conf\n -Dsun.security.krb5.debug=true\n volumes:\n - /tmp/kafka-config-test/secrets:/etc/kafka/secrets\n labels:\n - io.confluent.docker.testing=true\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/fixtures/standalone-network.yml",
"content": "---\nversion: '2'\nnetworks:\n zk:\n driver: bridge\nservices:\n zookeeper-bridge:\n image: confluentinc/cp-zookeeper:latest\n networks:\n - zk\n environment:\n ZOOKEEPER_CLIENT_PORT: 2181\n ZOOKEEPER_TICK_TIME: 2000\n ports:\n - 22181:2181\n - 32888:2888\n - 33888:3888\n labels:\n - io.confluent.docker.testing=true\n\n kafka-bridge:\n image: confluentinc/cp-kafka:latest\n networks:\n - zk\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper-bridge:2181\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:19092\n ports:\n - 19092:19092\n labels:\n - io.confluent.docker.testing=true\n\n zookeeper-host:\n image: confluentinc/cp-zookeeper:latest\n network_mode: host\n environment:\n ZOOKEEPER_CLIENT_PORT: 32181\n ZOOKEEPER_TICK_TIME: 2000\n labels:\n - io.confluent.docker.testing=true\n\n kafka-host:\n image: confluentinc/cp-kafka:latest\n network_mode: host\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: localhost:32181\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:29092\n labels:\n - io.confluent.docker.testing=true\n\n kafka-bridged-jmx:\n image: confluentinc/cp-kafka:latest\n networks:\n - zk\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper-bridge:2181/jmx\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:19092\n KAFKA_JMX_PORT: 9999\n ports:\n - 9999:9999\n labels:\n - io.confluent.docker.testing=true\n\n kafka-host-jmx:\n image: confluentinc/cp-kafka:latest\n network_mode: host\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: localhost:32181/jmx\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:39092\n KAFKA_JMX_PORT: 39999\n labels:\n - io.confluent.docker.testing=true\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/test/test_kafka.py",
"content": "import os\nimport unittest\nimport utils\nimport time\nimport string\nimport json\n\nCURRENT_DIR = os.path.dirname(os.path.abspath(__file__))\nFIXTURES_DIR = os.path.join(CURRENT_DIR, \"fixtures\", \"debian\", \"kafka\")\nHEALTH_CHECK = \"\"\"bash -c 'cp /etc/kafka/kafka.properties /tmp/cub.properties \\\n && echo security.protocol={security_protocol} >> /tmp/cub.properties \\\n && cub kafka-ready {brokers} 40 -b {host}:{port} -c /tmp/cub.properties -s {security_protocol}\\\n && echo PASS || echo FAIL'\n \"\"\"\nZK_READY = \"bash -c 'cub zk-ready {servers} 40 && echo PASS || echo FAIL'\"\nKAFKA_CHECK = \"bash -c 'kafkacat -L -b {host}:{port} -J' \"\nKAFKA_SASL_SSL_CHECK = \"\"\"bash -c \"kafkacat -X 'security.protocol=sasl_ssl' \\\n -X 'ssl.ca.location=/etc/kafka/secrets/snakeoil-ca-1.crt' \\\n -X 'ssl.certificate.location=/etc/kafka/secrets/kafkacat-ca1-signed.pem' \\\n -X 'ssl.key.location=/etc/kafka/secrets/kafkacat.client.key' \\\n -X 'ssl.key.password=confluent' \\\n -X 'sasl.kerberos.service.name={broker_principal}' \\\n -X 'sasl.kerberos.keytab=/etc/kafka/secrets/{client_principal}.keytab' \\\n -X 'sasl.kerberos.principal={client_principal}/{client_host}' \\\n -L -b {host}:{port} -J \"\n \"\"\"\n\nKAFKA_SSL_CHECK = \"\"\"kafkacat -X security.protocol=ssl \\\n -X ssl.ca.location=/etc/kafka/secrets/snakeoil-ca-1.crt \\\n -X ssl.certificate.location=/etc/kafka/secrets/kafkacat-ca1-signed.pem \\\n -X ssl.key.location=/etc/kafka/secrets/kafkacat.client.key \\\n -X ssl.key.password=confluent \\\n -L -b {host}:{port} -J\"\"\"\n\nKADMIN_KEYTAB_CREATE = \"\"\"bash -c \\\n 'kadmin.local -q \"addprinc -randkey {principal}/{hostname}@TEST.CONFLUENT.IO\" && \\\n kadmin.local -q \"ktadd -norandkey -k /tmp/keytab/{filename}.keytab {principal}/{hostname}@TEST.CONFLUENT.IO\"'\n \"\"\"\n\nPRODUCER = \"\"\"bash -c \"\\\n kafka-topics --create --topic {topic} --partitions 1 --replication-factor 3 --if-not-exists --zookeeper $KAFKA_ZOOKEEPER_CONNECT \\\n && seq {messages} | kafka-console-producer --broker-list {brokers} --topic {topic} --producer.config /etc/kafka/secrets/{config} \\\n && seq {messages} | kafka-console-producer --broker-list {brokers} --topic {topic} --producer.config /etc/kafka/secrets/{config} \\\n && echo PRODUCED {messages} messages.\"\n \"\"\"\n\nCONSUMER = \"\"\"bash -c \"\\\n export KAFKA_TOOLS_LOG4J_LOGLEVEL=DEBUG \\\n && dub template \"/etc/confluent/docker/tools-log4j.properties.template\" \"/etc/kafka/tools-log4j.properties\" \\\n && kafka-console-consumer --bootstrap-server {brokers} --topic foo --from-beginning --consumer.config /etc/kafka/secrets/{config} --max-messages {messages}\"\n \"\"\"\n\nKAFKACAT_SSL_CONSUMER = \"\"\"kafkacat -X security.protocol=ssl \\\n -X ssl.ca.location=/etc/kafka/secrets/snakeoil-ca-1.crt \\\n -X ssl.certificate.location=/etc/kafka/secrets/kafkacat-ca1-signed.pem \\\n -X ssl.key.location=/etc/kafka/secrets/kafkacat.client.key \\\n -X ssl.key.password=confluent \\\n -b {brokers} \\\n -C -t {topic} -c {messages}\n \"\"\"\n\nPLAIN_CLIENTS = \"\"\"bash -c \"\\\n export KAFKA_TOOLS_LOG4J_LOGLEVEL=DEBUG \\\n && dub template /etc/confluent/docker/tools-log4j.properties.template /etc/kafka/tools-log4j.properties \\\n && kafka-topics --create --topic {topic} --partitions 1 --replication-factor 3 --if-not-exists --zookeeper $KAFKA_ZOOKEEPER_CONNECT \\\n && seq {messages} | kafka-console-producer --broker-list {brokers} --topic {topic} \\\n && echo PRODUCED {messages} messages. \\\n && kafka-console-consumer --bootstrap-server {brokers} --topic foo --from-beginning --max-messages {messages}\"\n \"\"\"\n\nJMX_CHECK = \"\"\"bash -c \"\\\n echo 'get -b kafka.server:id=1,type=app-info Version' |\n java -jar jmxterm-1.0-alpha-4-uber.jar -l {jmx_hostname}:{jmx_port} -n -v silent \"\n\"\"\"\n\n\nclass ConfigTest(unittest.TestCase):\n\n @classmethod\n def setUpClass(cls):\n machine_name = os.environ[\"DOCKER_MACHINE_NAME\"]\n cls.machine = utils.TestMachine(machine_name)\n\n # Create directories with the correct permissions for test with userid and external volumes.\n cls.machine.ssh(\"mkdir -p /tmp/kafka-config-kitchen-sink-test/data\")\n cls.machine.ssh(\"sudo chown -R 12345 /tmp/kafka-config-kitchen-sink-test/data\")\n\n # Copy SSL files.\n cls.machine.ssh(\"mkdir -p /tmp/kafka-config-test/secrets\")\n local_secrets_dir = os.path.join(FIXTURES_DIR, \"secrets\")\n cls.machine.scp_to_machine(local_secrets_dir, \"/tmp/kafka-config-test\")\n\n cls.cluster = utils.TestCluster(\"config-test\", FIXTURES_DIR, \"standalone-config.yml\")\n cls.cluster.start()\n\n # Create keytabs\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"broker1\", principal=\"kafka\", hostname=\"sasl-ssl-config\"))\n\n assert \"PASS\" in cls.cluster.run_command_on_service(\"zookeeper\", ZK_READY.format(servers=\"localhost:2181\"))\n\n @classmethod\n def tearDownClass(cls):\n cls.cluster.shutdown()\n cls.machine.ssh(\"sudo rm -rf /tmp/kafka-config-kitchen-sink-test\")\n cls.machine.ssh(\"sudo rm -rf /tmp/kafka-config-test/secrets\")\n\n @classmethod\n def is_kafka_healthy_for_service(cls, service, port, num_brokers, host=\"localhost\", security_protocol=\"PLAINTEXT\"):\n output = cls.cluster.run_command_on_service(service, HEALTH_CHECK.format(host=host, port=port, brokers=num_brokers, security_protocol=security_protocol))\n assert \"PASS\" in output\n\n def test_required_config_failure(self):\n self.assertTrue(\"KAFKA_ZOOKEEPER_CONNECT is required.\" in self.cluster.service_logs(\"failing-config-zk-connect\", stopped=True))\n self.assertTrue(\"KAFKA_ADVERTISED_LISTENERS is required.\" in self.cluster.service_logs(\"failing-config-adv-listeners\", stopped=True))\n # Deprecated props.\n self.assertTrue(\"advertised.host is deprecated. Please use KAFKA_ADVERTISED_LISTENERS instead.\" in self.cluster.service_logs(\"failing-config-adv-hostname\", stopped=True))\n self.assertTrue(\"advertised.port is deprecated. Please use KAFKA_ADVERTISED_LISTENERS instead.\" in self.cluster.service_logs(\"failing-config-adv-port\", stopped=True))\n self.assertTrue(\"port is deprecated. Please use KAFKA_ADVERTISED_LISTENERS instead.\" in self.cluster.service_logs(\"failing-config-port\", stopped=True))\n self.assertTrue(\"host is deprecated. Please use KAFKA_ADVERTISED_LISTENERS instead.\" in self.cluster.service_logs(\"failing-config-host\", stopped=True))\n # SSL\n self.assertTrue(\"KAFKA_SSL_KEYSTORE_FILENAME is required.\" in self.cluster.service_logs(\"failing-config-ssl-keystore\", stopped=True))\n self.assertTrue(\"KAFKA_SSL_KEYSTORE_CREDENTIALS is required.\" in self.cluster.service_logs(\"failing-config-ssl-keystore-password\", stopped=True))\n self.assertTrue(\"KAFKA_SSL_KEY_CREDENTIALS is required.\" in self.cluster.service_logs(\"failing-config-ssl-key-password\", stopped=True))\n self.assertTrue(\"KAFKA_SSL_TRUSTSTORE_FILENAME is required.\" in self.cluster.service_logs(\"failing-config-ssl-truststore\", stopped=True))\n self.assertTrue(\"KAFKA_SSL_TRUSTSTORE_CREDENTIALS is required.\" in self.cluster.service_logs(\"failing-config-ssl-truststore-password\", stopped=True))\n\n self.assertTrue(\"KAFKA_OPTS is required.\" in self.cluster.service_logs(\"failing-config-sasl-jaas\", stopped=True))\n self.assertTrue(\"KAFKA_OPTS should contain 'java.security.auth.login.config' property.\" in self.cluster.service_logs(\"failing-config-sasl-missing-prop\", stopped=True))\n\n def test_default_config(self):\n self.is_kafka_healthy_for_service(\"default-config\", 9092, 1)\n props = self.cluster.run_command_on_service(\"default-config\", \"bash -c 'cat /etc/kafka/kafka.properties | sort'\")\n expected = \"\"\"\n advertised.listeners=PLAINTEXT://default-config:9092\n listeners=PLAINTEXT://0.0.0.0:9092\n log.dirs=/var/lib/kafka/data\n zookeeper.connect=zookeeper:2181/defaultconfig\n \"\"\"\n self.assertEquals(props.translate(None, string.whitespace), expected.translate(None, string.whitespace))\n\n logs = utils.run_docker_command(\n image=\"confluentinc/cp-kafkacat\",\n command=KAFKA_CHECK.format(host=\"default-config\", port=9092),\n host_config={'NetworkMode': 'config-test_default'})\n\n parsed_logs = json.loads(logs)\n expected_brokers = [{\"id\": 1001, \"name\": \"default-config:9092\"}]\n self.assertEquals(sorted(expected_brokers), sorted(parsed_logs[\"brokers\"]))\n\n def test_default_logging_config(self):\n self.is_kafka_healthy_for_service(\"default-config\", 9092, 1)\n\n log4j_props = self.cluster.run_command_on_service(\"default-config\", \"cat /etc/kafka/log4j.properties\")\n expected_log4j_props = \"\"\"log4j.rootLogger=INFO, stdout\n\n log4j.appender.stdout=org.apache.log4j.ConsoleAppender\n log4j.appender.stdout.layout=org.apache.log4j.PatternLayout\n log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n\n\n\n log4j.logger.kafka.authorizer.logger=WARN\n log4j.logger.kafka.log.LogCleaner=INFO\n log4j.logger.kafka.producer.async.DefaultEventHandler=DEBUG\n log4j.logger.kafka.controller=TRACE\n log4j.logger.kafka.network.RequestChannel$=WARN\n log4j.logger.kafka.request.logger=WARN\n log4j.logger.state.change.logger=TRACE\n log4j.logger.kafka=INFO\n \"\"\"\n self.assertEquals(log4j_props.translate(None, string.whitespace), expected_log4j_props.translate(None, string.whitespace))\n\n tools_log4j_props = self.cluster.run_command_on_service(\"default-config\", \"cat /etc/kafka/tools-log4j.properties\")\n expected_tools_log4j_props = \"\"\"log4j.rootLogger=WARN, stderr\n\n log4j.appender.stderr=org.apache.log4j.ConsoleAppender\n log4j.appender.stderr.layout=org.apache.log4j.PatternLayout\n log4j.appender.stderr.layout.ConversionPattern=[%d] %p %m (%c)%n\n log4j.appender.stderr.Target=System.err\n \"\"\"\n self.assertEquals(tools_log4j_props.translate(None, string.whitespace), expected_tools_log4j_props.translate(None, string.whitespace))\n\n def test_full_config(self):\n self.is_kafka_healthy_for_service(\"full-config\", 9092, 1)\n props = self.cluster.run_command_on_service(\"full-config\", \"bash -c 'cat /etc/kafka/kafka.properties | sort'\")\n expected = \"\"\"\n advertised.listeners=PLAINTEXT://full-config:9092\n broker.id=1\n listeners=PLAINTEXT://0.0.0.0:9092\n log.dirs=/var/lib/kafka/data\n zookeeper.connect=zookeeper:2181/fullconfig\n \"\"\"\n self.assertEquals(props.translate(None, string.whitespace), expected.translate(None, string.whitespace))\n\n def test_full_logging_config(self):\n self.is_kafka_healthy_for_service(\"full-config\", 9092, 1)\n\n log4j_props = self.cluster.run_command_on_service(\"full-config\", \"cat /etc/kafka/log4j.properties\")\n expected_log4j_props = \"\"\"log4j.rootLogger=WARN, stdout\n\n log4j.appender.stdout=org.apache.log4j.ConsoleAppender\n log4j.appender.stdout.layout=org.apache.log4j.PatternLayout\n log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n\n\n\n log4j.logger.kafka.authorizer.logger=WARN\n log4j.logger.kafka.log.LogCleaner=INFO\n log4j.logger.kafka.producer.async.DefaultEventHandler=DEBUG\n log4j.logger.kafka.controller=WARN\n log4j.logger.kafka.network.RequestChannel$=WARN\n log4j.logger.kafka.request.logger=WARN\n log4j.logger.state.change.logger=TRACE\n log4j.logger.kafka.foo.bar=DEBUG\n log4j.logger.kafka=INFO\n \"\"\"\n self.assertEquals(log4j_props.translate(None, string.whitespace), expected_log4j_props.translate(None, string.whitespace))\n\n tools_log4j_props = self.cluster.run_command_on_service(\"full-config\", \"cat /etc/kafka/tools-log4j.properties\")\n expected_tools_log4j_props = \"\"\"log4j.rootLogger=ERROR, stderr\n\n log4j.appender.stderr=org.apache.log4j.ConsoleAppender\n log4j.appender.stderr.layout=org.apache.log4j.PatternLayout\n log4j.appender.stderr.layout.ConversionPattern=[%d] %p %m (%c)%n\n log4j.appender.stderr.Target=System.err\n \"\"\"\n self.assertEquals(tools_log4j_props.translate(None, string.whitespace), expected_tools_log4j_props.translate(None, string.whitespace))\n\n def test_volumes(self):\n self.is_kafka_healthy_for_service(\"external-volumes\", 9092, 1)\n\n def test_random_user(self):\n self.is_kafka_healthy_for_service(\"random-user\", 9092, 1)\n\n def test_kitchen_sink(self):\n self.is_kafka_healthy_for_service(\"kitchen-sink\", 9092, 1)\n zk_props = self.cluster.run_command_on_service(\"kitchen-sink\", \"bash -c 'cat /etc/kafka/kafka.properties | sort'\")\n expected = \"\"\"\n advertised.listeners=PLAINTEXT://kitchen-sink:9092\n broker.id=1\n confluent.support.customer.id=c0\n confluent.support.metrics.enable=false\n listeners=PLAINTEXT://0.0.0.0:9092\n log.dirs=/var/lib/kafka/data\n zookeeper.connect=zookeeper:2181/kitchensink\n \"\"\"\n self.assertEquals(zk_props.translate(None, string.whitespace), expected.translate(None, string.whitespace))\n\n def test_ssl_config(self):\n self.is_kafka_healthy_for_service(\"ssl-config\", 9092, 1, \"ssl-config\", \"SSL\")\n zk_props = self.cluster.run_command_on_service(\"ssl-config\", \"bash -c 'cat /etc/kafka/kafka.properties | sort'\")\n expected = \"\"\"\n advertised.listeners=SSL://ssl-config:9092\n broker.id=1\n listeners=SSL://0.0.0.0:9092\n log.dirs=/var/lib/kafka/data\n security.inter.broker.protocol=SSL\n ssl.key.credentials=broker1_sslkey_creds\n ssl.key.password=confluent\n ssl.keystore.credentials=broker1_keystore_creds\n ssl.keystore.filename=kafka.broker1.keystore.jks\n ssl.keystore.location=/etc/kafka/secrets/kafka.broker1.keystore.jks\n ssl.keystore.password=confluent\n ssl.truststore.credentials=broker1_truststore_creds\n ssl.truststore.filename=kafka.broker1.truststore.jks\n ssl.truststore.location=/etc/kafka/secrets/kafka.broker1.truststore.jks\n ssl.truststore.password=confluent\n zookeeper.connect=zookeeper:2181/sslconfig\n \"\"\"\n self.assertEquals(zk_props.translate(None, string.whitespace), expected.translate(None, string.whitespace))\n\n def test_sasl_config(self):\n self.is_kafka_healthy_for_service(\"sasl-ssl-config\", 9094, 1, \"sasl-ssl-config\", \"SASL_SSL\")\n zk_props = self.cluster.run_command_on_service(\"sasl-ssl-config\", \"bash -c 'cat /etc/kafka/kafka.properties | sort'\")\n expected = \"\"\"\n advertised.listeners=SSL://sasl-ssl-config:9092,SASL_SSL://sasl-ssl-config:9094\n broker.id=1\n listeners=SSL://0.0.0.0:9092,SASL_SSL://0.0.0.0:9094\n log.dirs=/var/lib/kafka/data\n sasl.enabled.mechanisms=GSSAPI\n sasl.kerberos.service.name=kafka\n sasl.mechanism.inter.broker.protocol=GSSAPI\n security.inter.broker.protocol=SASL_SSL\n ssl.key.credentials=broker1_sslkey_creds\n ssl.key.password=confluent\n ssl.keystore.credentials=broker1_keystore_creds\n ssl.keystore.filename=kafka.broker1.keystore.jks\n ssl.keystore.location=/etc/kafka/secrets/kafka.broker1.keystore.jks\n ssl.keystore.password=confluent\n ssl.truststore.credentials=broker1_truststore_creds\n ssl.truststore.filename=kafka.broker1.truststore.jks\n ssl.truststore.location=/etc/kafka/secrets/kafka.broker1.truststore.jks\n ssl.truststore.password=confluent\n zookeeper.connect=zookeeper:2181/sslsaslconfig\n \"\"\"\n self.assertEquals(zk_props.translate(None, string.whitespace), expected.translate(None, string.whitespace))\n\n\nclass StandaloneNetworkingTest(unittest.TestCase):\n\n @classmethod\n def setUpClass(cls):\n cls.cluster = utils.TestCluster(\"standalone-network-test\", FIXTURES_DIR, \"standalone-network.yml\")\n cls.cluster.start()\n assert \"PASS\" in cls.cluster.run_command_on_service(\"zookeeper-bridge\", ZK_READY.format(servers=\"localhost:2181\"))\n assert \"PASS\" in cls.cluster.run_command_on_service(\"zookeeper-host\", ZK_READY.format(servers=\"localhost:32181\"))\n\n @classmethod\n def tearDownClass(cls):\n cls.cluster.shutdown()\n\n @classmethod\n def is_kafka_healthy_for_service(cls, service, port, num_brokers, host=\"localhost\", security_protocol=\"PLAINTEXT\"):\n output = cls.cluster.run_command_on_service(service, HEALTH_CHECK.format(host=host, port=port, brokers=num_brokers, security_protocol=security_protocol))\n assert \"PASS\" in output\n\n def test_bridged_network(self):\n # Test from within the container\n self.is_kafka_healthy_for_service(\"kafka-bridge\", 19092, 1)\n # Test from outside the container\n logs = utils.run_docker_command(\n image=\"confluentinc/cp-kafkacat\",\n command=KAFKA_CHECK.format(host=\"localhost\", port=19092),\n host_config={'NetworkMode': 'host'})\n\n parsed_logs = json.loads(logs)\n self.assertEquals(1, len(parsed_logs[\"brokers\"]))\n self.assertEquals(1, parsed_logs[\"brokers\"][0][\"id\"])\n self.assertEquals(\"localhost:19092\", parsed_logs[\"brokers\"][0][\"name\"])\n\n def test_host_network(self):\n # Test from within the container\n self.is_kafka_healthy_for_service(\"kafka-host\", 29092, 1)\n # Test from outside the container\n logs = utils.run_docker_command(\n image=\"confluentinc/cp-kafkacat\",\n command=KAFKA_CHECK.format(host=\"localhost\", port=29092),\n host_config={'NetworkMode': 'host'})\n\n parsed_logs = json.loads(logs)\n self.assertEquals(1, len(parsed_logs[\"brokers\"]))\n self.assertEquals(1, parsed_logs[\"brokers\"][0][\"id\"])\n self.assertEquals(\"localhost:29092\", parsed_logs[\"brokers\"][0][\"name\"])\n\n def test_jmx_host_network(self):\n\n # Test from outside the container\n logs = utils.run_docker_command(\n image=\"confluentinc/cp-jmxterm\",\n command=JMX_CHECK.format(jmx_hostname=\"localhost\", jmx_port=\"39999\"),\n host_config={'NetworkMode': 'host'})\n self.assertTrue(\"Version = 0.11.0.0-cp1;\" in logs)\n\n def test_jmx_bridged_network(self):\n\n # Test from outside the container\n logs = utils.run_docker_command(\n image=\"confluentinc/cp-jmxterm\",\n command=JMX_CHECK.format(jmx_hostname=\"kafka-bridged-jmx\", jmx_port=\"9999\"),\n host_config={'NetworkMode': 'standalone-network-test_zk'})\n self.assertTrue(\"Version = 0.11.0.0-cp1;\" in logs)\n\n\nclass ClusterBridgedNetworkTest(unittest.TestCase):\n @classmethod\n def setUpClass(cls):\n cls.cluster = utils.TestCluster(\"cluster-test\", FIXTURES_DIR, \"cluster-bridged-plain.yml\")\n cls.cluster.start()\n assert \"PASS\" in cls.cluster.run_command_on_service(\"zookeeper-1\", ZK_READY.format(servers=\"zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181\"))\n\n @classmethod\n def tearDownClass(cls):\n cls.cluster.shutdown()\n\n def test_cluster_running(self):\n self.assertTrue(self.cluster.is_running())\n\n @classmethod\n def is_kafka_healthy_for_service(cls, service, port, num_brokers, host=\"localhost\", security_protocol=\"PLAINTEXT\"):\n output = cls.cluster.run_command_on_service(service, HEALTH_CHECK.format(host=host, port=port, brokers=num_brokers, security_protocol=security_protocol))\n assert \"PASS\" in output\n\n def test_bridge_network(self):\n # Test from within the container\n self.is_kafka_healthy_for_service(\"kafka-1\", 9092, 3)\n # Test from outside the container\n logs = utils.run_docker_command(\n image=\"confluentinc/cp-kafkacat\",\n command=KAFKA_CHECK.format(host=\"kafka-1\", port=9092),\n host_config={'NetworkMode': 'cluster-test_zk'})\n\n parsed_logs = json.loads(logs)\n self.assertEquals(3, len(parsed_logs[\"brokers\"]))\n expected_brokers = [{\"id\": 1, \"name\": \"kafka-1:9092\"}, {\"id\": 2, \"name\": \"kafka-2:9092\"}, {\"id\": 3, \"name\": \"kafka-3:9092\"}]\n self.assertEquals(sorted(expected_brokers), sorted(parsed_logs[\"brokers\"]))\n\n client_logs = utils.run_docker_command(\n 300,\n image=\"confluentinc/cp-kafka\",\n name=\"kafka-producer\",\n environment={'KAFKA_ZOOKEEPER_CONNECT': \"zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181\"},\n command=PLAIN_CLIENTS.format(brokers=\"kafka-1:9092\", topic=\"foo\", messages=100),\n host_config={'NetworkMode': 'cluster-test_zk'})\n\n self.assertTrue(\"Processed a total of 100 messages\" in client_logs)\n\n\nclass ClusterSSLBridgedNetworkTest(ClusterBridgedNetworkTest):\n @classmethod\n def setUpClass(cls):\n machine_name = os.environ[\"DOCKER_MACHINE_NAME\"]\n cls.machine = utils.TestMachine(machine_name)\n\n # Copy SSL files.\n print cls.machine.ssh(\"mkdir -p /tmp/kafka-cluster-bridge-test/secrets\")\n local_secrets_dir = os.path.join(FIXTURES_DIR, \"secrets\")\n cls.machine.scp_to_machine(local_secrets_dir, \"/tmp/kafka-cluster-bridge-test\")\n\n cls.cluster = utils.TestCluster(\"cluster-test\", FIXTURES_DIR, \"cluster-bridged-ssl.yml\")\n cls.cluster.start()\n assert \"PASS\" in cls.cluster.run_command_on_service(\"zookeeper-1\", ZK_READY.format(servers=\"zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181\"))\n\n @classmethod\n def tearDownClass(cls):\n cls.cluster.shutdown()\n cls.machine.ssh(\"sudo rm -rf /tmp/kafka-cluster-bridge-test/secrets\")\n\n def test_bridge_network(self):\n # Test from within the container\n self.is_kafka_healthy_for_service(\"kafka-ssl-1\", 9093, 3, \"kafka-ssl-1\", \"SSL\")\n # Test from outside the container\n logs = utils.run_docker_command(\n image=\"confluentinc/cp-kafkacat\",\n command=KAFKA_SSL_CHECK.format(host=\"kafka-ssl-1\", port=9093),\n host_config={'NetworkMode': 'cluster-test_zk', 'Binds': ['/tmp/kafka-cluster-bridge-test/secrets:/etc/kafka/secrets']})\n\n parsed_logs = json.loads(logs)\n self.assertEquals(3, len(parsed_logs[\"brokers\"]))\n expected_brokers = [{\"id\": 1, \"name\": \"kafka-ssl-1:9093\"}, {\"id\": 2, \"name\": \"kafka-ssl-2:9093\"}, {\"id\": 3, \"name\": \"kafka-ssl-3:9093\"}]\n self.assertEquals(sorted(expected_brokers), sorted(parsed_logs[\"brokers\"]))\n\n producer_logs = utils.run_docker_command(\n 300,\n image=\"confluentinc/cp-kafka\",\n name=\"kafka-ssl-bridged-producer\",\n environment={'KAFKA_ZOOKEEPER_CONNECT': \"zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/ssl\"},\n command=PRODUCER.format(brokers=\"kafka-ssl-1:9093\", topic=\"foo\", config=\"bridged.producer.ssl.config\", messages=100),\n host_config={'NetworkMode': 'cluster-test_zk', 'Binds': ['/tmp/kafka-cluster-bridge-test/secrets:/etc/kafka/secrets']})\n\n self.assertTrue(\"PRODUCED 100 messages\" in producer_logs)\n\n consumer_logs = utils.run_docker_command(\n 300,\n image=\"confluentinc/cp-kafkacat\",\n name=\"kafkacat-ssl-bridged-consumer\",\n command=KAFKACAT_SSL_CONSUMER.format(brokers=\"kafka-ssl-1:9093\", topic=\"foo\", messages=10),\n host_config={'NetworkMode': 'cluster-test_zk', 'Binds': ['/tmp/kafka-cluster-bridge-test/secrets:/etc/kafka/secrets']})\n\n self.assertEquals(\"\\n\".join([str(i + 1) for i in xrange(10)]), consumer_logs.strip())\n\n\nclass ClusterSASLBridgedNetworkTest(ClusterBridgedNetworkTest):\n @classmethod\n def setUpClass(cls):\n machine_name = os.environ[\"DOCKER_MACHINE_NAME\"]\n cls.machine = utils.TestMachine(machine_name)\n\n # Copy SSL files.\n print cls.machine.ssh(\"mkdir -p /tmp/kafka-cluster-bridge-test/secrets\")\n local_secrets_dir = os.path.join(FIXTURES_DIR, \"secrets\")\n cls.machine.scp_to_machine(local_secrets_dir, \"/tmp/kafka-cluster-bridge-test\")\n\n cls.cluster = utils.TestCluster(\"cluster-test\", FIXTURES_DIR, \"cluster-bridged-sasl.yml\")\n cls.cluster.start()\n\n # Create keytabs\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"bridged_broker1\", principal=\"kafka\", hostname=\"kafka-sasl-ssl-1\"))\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"bridged_broker2\", principal=\"kafka\", hostname=\"kafka-sasl-ssl-2\"))\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"bridged_broker3\", principal=\"kafka\", hostname=\"kafka-sasl-ssl-3\"))\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"bridged_kafkacat\", principal=\"bridged_kafkacat\", hostname=\"bridged-kafkacat\"))\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"bridged_producer\", principal=\"bridged_producer\", hostname=\"kafka-sasl-ssl-producer\"))\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"bridged_consumer\", principal=\"bridged_consumer\", hostname=\"kafka-sasl-ssl-consumer\"))\n\n assert \"PASS\" in cls.cluster.run_command_on_service(\"zookeeper-1\", ZK_READY.format(servers=\"zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181\"))\n\n @classmethod\n def tearDownClass(cls):\n cls.cluster.shutdown()\n cls.machine.ssh(\"sudo rm -rf /tmp/kafka-cluster-bridge-test/secrets\")\n\n def test_bridge_network(self):\n # Test from within the container\n self.is_kafka_healthy_for_service(\"kafka-sasl-ssl-1\", 9094, 3, \"kafka-sasl-ssl-1\", \"SASL_SSL\")\n\n # FIXME: Figure out how to use kafkacat with SASL/Kerberos\n # Test from outside the container\n # logs = utils.run_docker_command(\n # image=\"confluentinc/cp-kafkacat\",\n # name=\"bridged-kafkacat\",\n # command=KAFKA_SASL_SSL_CHECK.format(host=\"kafka-sasl-ssl-1\", port=9094, broker_principal=\"kafka\", client_principal=\"bridged_kafkacat\", client_host=\"bridged-kafkacat\"),\n # host_config={'NetworkMode': 'cluster-test_zk', 'Binds': ['/tmp/kafka-cluster-bridge-test/secrets:/etc/kafka/secrets', '/tmp/kafka-cluster-bridge-test/secrets/bridged_krb.conf:/etc/krb5.conf']})\n #\n # parsed_logs = json.loads(logs)\n # self.assertEquals(3, len(parsed_logs[\"brokers\"]))\n # expected_brokers = [{\"id\": 1, \"name\": \"kafka-sasl-ssl-1:9094\"}, {\"id\": 2, \"name\": \"kafka-sasl-ssl-2:9094\"}, {\"id\": 3, \"name\": \"kafka-sasl-ssl-3:9094\"}]\n # self.assertEquals(sorted(expected_brokers), sorted(parsed_logs[\"brokers\"]))\n\n producer_env = {'KAFKA_ZOOKEEPER_CONNECT': \"zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/saslssl\",\n 'KAFKA_OPTS': \"-Djava.security.auth.login.config=/etc/kafka/secrets/bridged_producer_jaas.conf -Djava.security.krb5.conf=/etc/kafka/secrets/bridged_krb.conf -Dsun.net.spi.nameservice.provider.1=sun -Dsun.security.krb5.debug=true\"}\n producer_logs = utils.run_docker_command(\n 300,\n image=\"confluentinc/cp-kafka\",\n name=\"kafka-sasl-ssl-bridged-producer\",\n environment=producer_env,\n command=PRODUCER.format(brokers=\"kafka-sasl-ssl-1:9094\", topic=\"foo\", config=\"bridged.producer.ssl.sasl.config\", messages=100),\n host_config={'NetworkMode': 'cluster-test_zk', 'Binds': ['/tmp/kafka-cluster-bridge-test/secrets:/etc/kafka/secrets']})\n\n self.assertTrue(\"PRODUCED 100 messages\" in producer_logs)\n\n consumer_env = {'KAFKA_ZOOKEEPER_CONNECT': \"zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/saslssl\",\n 'KAFKA_OPTS': \"-Djava.security.auth.login.config=/etc/kafka/secrets/bridged_consumer_jaas.conf -Djava.security.krb5.conf=/etc/kafka/secrets/bridged_krb.conf -Dsun.net.spi.nameservice.provider.1=sun -Dsun.security.krb5.debug=true\"}\n\n consumer_logs = utils.run_docker_command(\n 300,\n image=\"confluentinc/cp-kafka\",\n name=\"kafka-sasl-ssl-bridged-consumer\",\n environment=consumer_env,\n command=CONSUMER.format(brokers=\"kafka-sasl-ssl-1:9094\", topic=\"foo\", config=\"bridged.consumer.ssl.sasl.config\", messages=10),\n host_config={'NetworkMode': 'cluster-test_zk', 'Binds': ['/tmp/kafka-cluster-bridge-test/secrets:/etc/kafka/secrets']})\n\n self.assertTrue(\"Processed a total of 10 messages\" in consumer_logs)\n\n\nclass ClusterHostNetworkTest(unittest.TestCase):\n @classmethod\n def setUpClass(cls):\n cls.cluster = utils.TestCluster(\"cluster-test\", FIXTURES_DIR, \"cluster-host-plain.yml\")\n cls.cluster.start()\n assert \"PASS\" in cls.cluster.run_command_on_service(\"zookeeper-1\", ZK_READY.format(servers=\"localhost:22181,localhost:32181,localhost:42181\"))\n\n @classmethod\n def tearDownClass(cls):\n cls.cluster.shutdown()\n\n def test_cluster_running(self):\n self.assertTrue(self.cluster.is_running())\n\n @classmethod\n def is_kafka_healthy_for_service(cls, service, port, num_brokers, host=\"localhost\", security_protocol=\"PLAINTEXT\"):\n output = cls.cluster.run_command_on_service(service, HEALTH_CHECK.format(host=host, port=port, brokers=num_brokers, security_protocol=security_protocol))\n assert \"PASS\" in output\n\n def test_host_network(self):\n # Test from within the container\n self.is_kafka_healthy_for_service(\"kafka-1\", 19092, 3)\n # Test from outside the container\n logs = utils.run_docker_command(\n image=\"confluentinc/cp-kafkacat\",\n command=KAFKA_CHECK.format(host=\"localhost\", port=19092),\n host_config={'NetworkMode': 'host'})\n\n parsed_logs = json.loads(logs)\n self.assertEquals(3, len(parsed_logs[\"brokers\"]))\n expected_brokers = [{\"id\": 1, \"name\": \"localhost:19092\"}, {\"id\": 2, \"name\": \"localhost:29092\"}, {\"id\": 3, \"name\": \"localhost:39092\"}]\n self.assertEquals(sorted(expected_brokers), sorted(parsed_logs[\"brokers\"]))\n\n client_logs = utils.run_docker_command(\n 300,\n image=\"confluentinc/cp-kafka\",\n name=\"kafka-producer\",\n environment={'KAFKA_ZOOKEEPER_CONNECT': \"localhost:22181,localhost:32181,localhost:42181\"},\n command=PLAIN_CLIENTS.format(brokers=\"localhost:19092\", topic=\"foo\", messages=100),\n host_config={'NetworkMode': 'host'})\n\n self.assertTrue(\"Processed a total of 100 messages\" in client_logs)\n\n\nclass ClusterSSLHostNetworkTest(ClusterHostNetworkTest):\n @classmethod\n def setUpClass(cls):\n machine_name = os.environ[\"DOCKER_MACHINE_NAME\"]\n cls.machine = utils.TestMachine(machine_name)\n\n # Copy SSL files.\n print cls.machine.ssh(\"mkdir -p /tmp/kafka-cluster-host-test/secrets\")\n local_secrets_dir = os.path.join(FIXTURES_DIR, \"secrets\")\n cls.machine.scp_to_machine(local_secrets_dir, \"/tmp/kafka-cluster-host-test\")\n\n cls.cluster = utils.TestCluster(\"cluster-test\", FIXTURES_DIR, \"cluster-host-ssl.yml\")\n cls.cluster.start()\n\n assert \"PASS\" in cls.cluster.run_command_on_service(\"zookeeper-1\", ZK_READY.format(servers=\"localhost:22181,localhost:32181,localhost:42181\"))\n\n @classmethod\n def tearDownClass(cls):\n cls.cluster.shutdown()\n cls.machine.ssh(\"sudo rm -rf /tmp/kafka-cluster-host-test/secrets\")\n\n def test_host_network(self):\n # Test from within the container\n self.is_kafka_healthy_for_service(\"kafka-ssl-1\", 19093, 3, \"localhost\", \"SSL\")\n # Test from outside the container\n logs = utils.run_docker_command(\n image=\"confluentinc/cp-kafkacat\",\n command=KAFKA_SSL_CHECK.format(host=\"localhost\", port=19093),\n host_config={'NetworkMode': 'host', 'Binds': ['/tmp/kafka-cluster-host-test/secrets:/etc/kafka/secrets']})\n\n parsed_logs = json.loads(logs)\n self.assertEquals(3, len(parsed_logs[\"brokers\"]))\n expected_brokers = [{\"id\": 1, \"name\": \"localhost:19093\"}, {\"id\": 2, \"name\": \"localhost:29093\"}, {\"id\": 3, \"name\": \"localhost:39093\"}]\n self.assertEquals(sorted(expected_brokers), sorted(parsed_logs[\"brokers\"]))\n\n producer_logs = utils.run_docker_command(\n 300,\n image=\"confluentinc/cp-kafka\",\n name=\"kafka-ssl-host-producer\",\n environment={'KAFKA_ZOOKEEPER_CONNECT': \"localhost:22181,localhost:32181,localhost:42181/ssl\"},\n command=PRODUCER.format(brokers=\"localhost:29093\", topic=\"foo\", config=\"host.producer.ssl.config\", messages=100),\n host_config={'NetworkMode': 'host', 'Binds': ['/tmp/kafka-cluster-host-test/secrets:/etc/kafka/secrets']})\n\n self.assertTrue(\"PRODUCED 100 messages\" in producer_logs)\n\n consumer_logs = utils.run_docker_command(\n 300,\n image=\"confluentinc/cp-kafkacat\",\n name=\"kafkacat-ssl-host-consumer\",\n command=KAFKACAT_SSL_CONSUMER.format(brokers=\"localhost:29093\", topic=\"foo\", messages=10),\n host_config={'NetworkMode': 'host', 'Binds': ['/tmp/kafka-cluster-host-test/secrets:/etc/kafka/secrets']})\n\n self.assertEquals(\"\\n\".join([str(i + 1) for i in xrange(10)]), consumer_logs.strip())\n\n\nclass ClusterSASLHostNetworkTest(ClusterHostNetworkTest):\n @classmethod\n def setUpClass(cls):\n machine_name = os.environ[\"DOCKER_MACHINE_NAME\"]\n cls.machine = utils.TestMachine(machine_name)\n\n # Add a hostname mapped to eth0, required for SASL to work predictably.\n # localhost and hostname both resolve to 127.0.0.1 in the docker image, so using localhost causes unprodicatable behaviour\n # with zkclient\n cmd = \"\"\"\n \"sudo sh -c 'grep sasl.kafka.com /etc/hosts || echo {IP} sasl.kafka.com >> /etc/hosts'\"\n \"\"\".strip()\n cls.machine.ssh(cmd.format(IP=cls.machine.get_internal_ip().strip()))\n\n # Copy SSL files.\n cls.machine.ssh(\"mkdir -p /tmp/kafka-cluster-host-test/secrets\")\n\n local_secrets_dir = os.path.join(FIXTURES_DIR, \"secrets\")\n cls.machine.scp_to_machine(local_secrets_dir, \"/tmp/kafka-cluster-host-test\")\n\n cls.cluster = utils.TestCluster(\"cluster-test\", FIXTURES_DIR, \"cluster-host-sasl.yml\")\n cls.cluster.start()\n\n # Create keytabs\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"host_broker1\", principal=\"kafka\", hostname=\"sasl.kafka.com\"))\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"host_broker2\", principal=\"kafka\", hostname=\"sasl.kafka.com\"))\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"host_broker3\", principal=\"kafka\", hostname=\"sasl.kafka.com\"))\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"host_producer\", principal=\"host_producer\", hostname=\"sasl.kafka.com\"))\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"host_consumer\", principal=\"host_consumer\", hostname=\"sasl.kafka.com\"))\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"zookeeper-host-1\", principal=\"zookeeper\", hostname=\"sasl.kafka.com\"))\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"zookeeper-host-2\", principal=\"zookeeper\", hostname=\"sasl.kafka.com\"))\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"zookeeper-host-3\", principal=\"zookeeper\", hostname=\"sasl.kafka.com\"))\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"zkclient-host-1\", principal=\"zkclient\", hostname=\"sasl.kafka.com\"))\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"zkclient-host-2\", principal=\"zkclient\", hostname=\"sasl.kafka.com\"))\n cls.cluster.run_command_on_service(\"kerberos\", KADMIN_KEYTAB_CREATE.format(filename=\"zkclient-host-3\", principal=\"zkclient\", hostname=\"sasl.kafka.com\"))\n\n assert \"PASS\" in cls.cluster.run_command_on_service(\"zookeeper-sasl-1\", ZK_READY.format(servers=\"sasl.kafka.com:22181,sasl.kafka.com:32181,sasl.kafka.com:42181\"))\n\n @classmethod\n def tearDownClass(cls):\n cls.cluster.shutdown()\n cls.machine.ssh(\"sudo rm -rf /tmp/kafka-cluster-host-test/secrets\")\n\n def test_host_network(self):\n # Test from within the container\n self.is_kafka_healthy_for_service(\"kafka-sasl-ssl-1\", 19094, 3, \"sasl.kafka.com\", \"SASL_SSL\")\n\n producer_env = {'KAFKA_ZOOKEEPER_CONNECT': \"sasl.kafka.com:22181,sasl.kafka.com:32181,sasl.kafka.com:42181/saslssl\",\n 'KAFKA_OPTS': \"-Djava.security.auth.login.config=/etc/kafka/secrets/host_producer_jaas.conf -Djava.security.krb5.conf=/etc/kafka/secrets/host_krb.conf -Dsun.net.spi.nameservice.provider.1=sun -Dsun.security.krb5.debug=true\"}\n producer_logs = utils.run_docker_command(\n 300,\n image=\"confluentinc/cp-kafka\",\n name=\"kafka-ssl-sasl-host-producer\",\n environment=producer_env,\n command=PRODUCER.format(brokers=\"sasl.kafka.com:29094\", topic=\"foo\", config=\"host.producer.ssl.sasl.config\", messages=100),\n host_config={'NetworkMode': 'host', 'Binds': ['/tmp/kafka-cluster-host-test/secrets:/etc/kafka/secrets']})\n\n self.assertTrue(\"PRODUCED 100 messages\" in producer_logs)\n\n consumer_env = {'KAFKA_ZOOKEEPER_CONNECT': \"sasl.kafka.com:22181,sasl.kafka.com:32181,sasl.kafka.com:42181/saslssl\",\n 'KAFKA_OPTS': \"-Djava.security.auth.login.config=/etc/kafka/secrets/host_consumer_jaas.conf -Djava.security.krb5.conf=/etc/kafka/secrets/host_krb.conf -Dsun.net.spi.nameservice.provider.1=sun -Dsun.security.krb5.debug=true\"}\n\n consumer_logs = utils.run_docker_command(\n 300,\n image=\"confluentinc/cp-kafka\",\n name=\"kafka-ssl-sasl-host-consumer\",\n environment=consumer_env,\n command=CONSUMER.format(brokers=\"sasl.kafka.com:29094\", topic=\"foo\", config=\"host.consumer.ssl.sasl.config\", messages=10),\n host_config={'NetworkMode': 'host', 'Binds': ['/tmp/kafka-cluster-host-test/secrets:/etc/kafka/secrets']})\n\n self.assertTrue(\"Processed a total of 10 messages\" in consumer_logs)\n"
},
{
"path": "kraft/none/image/kafka-images/kafka/tox.ini",
"content": "[tox]\nenvlist = test\ntoxworkdir = /var/tmp\n\n[testenv]\n# Consolidating all deps here instead of cleanly/separately in test/style/cover so we\n# have a single env (platform) to work with, which makes debugging easier (like which env?).\n# Not as clean but easier to work with for dev, which is better.\ndeps =\n -rrequirements.txt\n flake8\n pytest\n pytest-xdist\n pytest-cov\ninstall_command = pip install -U {packages}\nrecreate = True\nskipsdist = True\nusedevelop = True\nsetenv =\n PIP_PROCESS_DEPENDENCY_LINKS=1\n PIP_DEFAULT_TIMEOUT=60\n ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future\nbasepython = python3\nenvdir = {toxworkdir}/confluent\n\n[testenv:test]\ncommands =\n py.test --color=no {env:PYTESTARGS:} test\n\n[testenv:style]\ncommands =\n flake8 --config tox.ini\n\n[testenv:cover]\ncommands =\n py.test {env:PYTESTARGS:} --cov . --cov-report=xml --cov-report=html --cov-report=term test\n\n[flake8]\nignore = E111,E121,W292,E123,E226\nmax-line-length = 160\n\n[pytest]\naddopts = -n 1\n"
},
{
"path": "kraft/none/up",
"content": "#!/bin/sh\n\ndocker-compose up -d\n\n# Creating the user kafka\n# kafka is configured as a super user, no need for additional ACL\n# docker-compose exec kafka kafka-configs --zookeeper zookeeper:2181 --alter --add-config 'SCRAM-SHA-255=[password=kafka],SCRAM-SHA-512=[password=kafka]' --entity-type users --entity-name kafka\n\necho \"Example configuration:\"\necho \"-> docker-compose exec kafka kafka-console-producer --broker-list kafka:9092 --topic test\"\necho \"-> docker-compose exec kafka kafka-console-consumer --bootstrap-server kafka:9092 --topic test --from-beginning\"\n"
},
{
"path": "ldap/acls/acls.csv",
"content": "KafkaPrincipal,ResourceType,PatternType,ResourceName,Operation,PermissionType,Host\nUser:kafka,Cluster,LITERAL,kafka-cluster,All,Allow,*\nGroup:Kafka Developers,Group,LITERAL,*,Read,Allow,*\nGroup:Kafka Developers,Topic,LITERAL,test-topic,Describe,Allow,*\nGroup:Kafka Developers,Topic,LITERAL,test-topic,Read,Allow,*\nGroup:Kafka Developers,Topic,LITERAL,test-topic,Write,Allow,*\nGroup:Kafka Developers,Topic,LITERAL,test-topic,Create,Allow,*\n"
},
{
"path": "ldap/add-user",
"content": "#!/bin/sh\n\n# Creating the users\ndocker-compose exec kafka kafka-configs --zookeeper zookeeper:2181 --alter --add-config 'SCRAM-SHA-256=[password=purbon-secret],SCRAM-SHA-512=[password=purbon-secret]' --entity-type users --entity-name purbon\n\n\necho \"Should succeed as the new user is in the group\"\necho \"-> docker-compose exec kafka kafka-console-producer --broker-list kafka:9093 --topic test-topic --producer.config=/service/kafka/users/purbon.properties\"\necho \"-> docker-compose exec kafka kafka-console-consumer --bootstrap-server kafka:9093 --consumer.config /service/kafka/users/purbon.properties --topic test-topic --from-beginning\"\n"
},
{
"path": "ldap/custom/01_base.ldif",
"content": "dn: ou=users,{{ LDAP_BASE_DN }}\nobjectClass: organizationalUnit\nou: Users\n\ndn: ou=groups,{{ LDAP_BASE_DN }}\nobjectClass: organizationalUnit\nou: Groups\n\n"
},
{
"path": "ldap/custom/02_KafkaDevelopers.ldif",
"content": "dn: cn=Kafka Developers,ou=groups,{{ LDAP_BASE_DN }}\nobjectClass: top\nobjectClass: posixGroup\ncn: Kafka Developers\ngidNumber: 5000\n\n"
},
{
"path": "ldap/custom/10_alice.ldif",
"content": "dn: cn=alice,ou=users,{{ LDAP_BASE_DN }}\nobjectClass: inetOrgPerson\nobjectClass: posixAccount\nobjectClass: shadowAccount\nuid: alice\nsn: LookingGlass\ngivenName: Alice\ncn: alice\ndisplayName: Alice LookingGlass\nuidNumber: 10000\ngidNumber: 5000\nuserPassword: alice-secret\ngecos: alice\nloginShell: /bin/bash\nhomeDirectory: /home/alice\n"
},
{
"path": "ldap/custom/11_barnie.ldif",
"content": "dn: cn=barnie,ou=users,{{ LDAP_BASE_DN }}\nobjectClass: inetOrgPerson\nobjectClass: posixAccount\nobjectClass: shadowAccount\nuid: barnie\nsn: Rubble\ngivenName: Barnie\ncn: barnie\ndisplayName: Barnie Rubble\nuidNumber: 10001\ngidNumber: 5000\nuserPassword: barnie-secret\ngecos: barnie\nloginShell: /bin/bash\nhomeDirectory: /home/barnie\n"
},
{
"path": "ldap/custom/12_charlie.ldif",
"content": "dn: cn=charlie,ou=users,{{ LDAP_BASE_DN }}\nobjectClass: inetOrgPerson\nobjectClass: posixAccount\nobjectClass: shadowAccount\nuid: charlie\nsn: Sheen\ngivenName: Charlie\ncn: charlie\ndisplayName: Charlie Sheen\nuidNumber: 10001\ngidNumber: 5000\nuserPassword: charlie-secret\ngecos: charlie\nloginShell: /bin/bash\nhomeDirectory: /home/charlie\n"
},
{
"path": "ldap/custom/20_group_add.ldif",
"content": "dn: cn=Kafka Developers,ou=groups,{{ LDAP_BASE_DN }}\nchangetype: modify\nadd: memberuid\nmemberuid: cn=alice,ou=users,{{ LDAP_BASE_DN }}\n\ndn: cn=Kafka Developers,ou=groups,{{ LDAP_BASE_DN }}\nchangetype: modify\nadd: memberuid\nmemberuid: cn=barnie,ou=users,{{ LDAP_BASE_DN }}\n"
},
{
"path": "ldap/docker-compose-with-ssl.yaml",
"content": "version: '3'\nservices:\n ldap:\n image: osixia/openldap:1.2.3\n hostname: ldap\n container_name: ldap\n environment:\n LDAP_ORGANISATION: \"Confluent\"\n LDAP_DOMAIN: \"confluent.io\"\n LDAP_TLS: \"true\"\n LDAP_TLS_CRT_FILENAME: my-ldap.crt\n LDAP_TLS_KEY_FILENAME: my-ldap.key\n LDAP_TLS_CA_CRT_FILENAME: my-ca.crt\n LDAP_TLS_VERIFY_CLIENT: demand\n ports:\n - \"389:389\"\n - \"636:636\"\n volumes:\n - \"$PWD/ldap/custom:/container/service/slapd/assets/config/bootstrap/ldif/custom\"\n - \"$PWD/ldap/certs:/container/service/slapd/assets/certs\"\n command: \"--copy-service\"\n\n zookeeper:\n build: zookeeper/\n hostname: zookeeper\n container_name: zookeeper\n environment:\n - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/zookeeper_server_jaas.conf\n\n kafka:\n build: kafka/\n container_name: kafka\n environment:\n - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf\n depends_on:\n - zookeeper\n - ldap\n volumes:\n - \"$PWD/kafka/jks:/etc/kafka/jks\"\n ports:\n - \"9093:9093\"\n command: [\"kafka-server-start\", \"/etc/kafka/server-with-ssl.properties\"]\n\n kafka-security-manager:\n image: simplesteph/kafka-security-manager:latest\n hostname: kafka-security-manager\n container_name: kafka-security-manager\n environment:\n - AUTHORIZER_ZOOKEEPER_CONNECT=zookeeper:2181\n - KSM_READONLY=false\n - SOURCE_CLASS=com.github.simplesteph.ksm.source.FileSourceAcl\n - SOURCE_FILE_FILENAME=/acls/acls.csv\n depends_on:\n - kafka\n volumes:\n - \"$PWD/acls:/acls\"\n\n"
},
{
"path": "ldap/docker-compose.yaml",
"content": "version: '3'\nservices:\n ldap:\n image: osixia/openldap:1.3.0\n hostname: ldap\n container_name: ldap\n environment:\n LDAP_ORGANISATION: \"Confluent\"\n LDAP_DOMAIN: \"confluent.io\"\n ports:\n - \"389:389\"\n - \"636:636\"\n volumes:\n - \"$PWD/ldap/custom:/container/service/slapd/assets/config/bootstrap/ldif/custom\"\n command: \"--copy-service\"\n\n phpldapadmin-service:\n image: osixia/phpldapadmin:0.9.0\n container_name: ldapadmin-service\n environment:\n - PHPLDAPADMIN_LDAP_HOSTS=ldap\n ports:\n - \"6444:443\"\n depends_on:\n - ldap\n\n zookeeper:\n build: zookeeper/\n hostname: zookeeper\n container_name: zookeeper\n environment:\n - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/zookeeper_server_jaas.conf\n\n kafka:\n build: kafka/\n container_name: kafka\n environment:\n - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf\n depends_on:\n - zookeeper\n - ldap\n volumes:\n - \"$PWD/kafka/users:/service/kafka/users\"\n - \"$PWD/kafka/jks:/etc/kafka/jks\"\n ports:\n - \"9093:9093\"\n command: [\"kafka-server-start\", \"/etc/kafka/server.properties\"]\n\n kafka-security-manager:\n image: simplesteph/kafka-security-manager:latest\n hostname: kafka-security-manager\n container_name: kafka-security-manager\n environment:\n - AUTHORIZER_ZOOKEEPER_CONNECT=zookeeper:2181\n - KSM_READONLY=false\n - SOURCE_CLASS=com.github.simplesteph.ksm.source.FileSourceAcl\n - SOURCE_FILE_FILENAME=/acls/acls.csv\n depends_on:\n - kafka\n volumes:\n - \"$PWD/acls:/acls\"\n\n"
},
{
"path": "ldap/kafka/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER seknop@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.5/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-11-openjdk\nRUN yum install -y confluent-server\nRUN yum install -y confluent-security\n\n# 3. Configure Kafka and zookeeper for Kerberos\nCOPY server.properties /etc/kafka/server.properties\nCOPY server-with-ssl.properties /etc/kafka/server-with-ssl.properties\nCOPY kafka.jaas.config /etc/kafka/kafka_server_jaas.conf\nCOPY log4j.properties /etc/kafka/log4j.properties\n\nCOPY alice.properties /etc/kafka/alice.properties\nCOPY barnie.properties /etc/kafka/barnie.properties\nCOPY charlie.properties /etc/kafka/charlie.properties\nCOPY kafka.properties /etc/kafka/kafka.properties\n\nEXPOSE 9093\n\nCMD kafka-server-start /etc/kafka/server.properties\n"
},
{
"path": "ldap/kafka/alice.properties",
"content": "sasl.mechanism=SCRAM-SHA-256\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"alice\" \\\n password=\"alice-secret\";\n"
},
{
"path": "ldap/kafka/barnie.properties",
"content": "sasl.mechanism=SCRAM-SHA-256\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"barnie\" \\\n password=\"barnie-secret\";\n"
},
{
"path": "ldap/kafka/charlie.properties",
"content": "sasl.mechanism=SCRAM-SHA-256\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"charlie\" \\\n password=\"charlie-secret\";\n"
},
{
"path": "ldap/kafka/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.5/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.5/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.5\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.5/archive.key\nenabled=1\n\n"
},
{
"path": "ldap/kafka/consumer.properties",
"content": "sasl.mechanism=SCRAM-SHA-256\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"kafka\" \\\n password=\"kafka\";\n\n"
},
{
"path": "ldap/kafka/jks/.gitignore",
"content": "*.crt\n*.csr\n*_creds\n*.jks\n*.srl\n*.key\n*.pem\n*.der\n*.p12\n"
},
{
"path": "ldap/kafka/kafka.jaas.config",
"content": "KafkaServer {\n org.apache.kafka.common.security.scram.ScramLoginModule required\n username=\"kafka\"\n password=\"kafka\";\n};\n\nClient {\n org.apache.zookeeper.server.auth.DigestLoginModule required\n username=\"kafka\"\n password=\"kafka\";\n};\n"
},
{
"path": "ldap/kafka/kafka.properties",
"content": "sasl.mechanism=SCRAM-SHA-256\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"kafka\" \\\n password=\"kafka\";\n"
},
{
"path": "ldap/kafka/log4j.properties",
"content": "# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# Unspecified loggers and loggers with additivity=true output to server.log and stdout\n# Note that INFO only applies to unspecified loggers, the log level of the child logger is used otherwise\n# Sven is here!\nlog4j.rootLogger=INFO, stdout, kafkaAppender\n\nlog4j.appender.stdout=org.apache.log4j.ConsoleAppender\nlog4j.appender.stdout.layout=org.apache.log4j.PatternLayout\nlog4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender\nlog4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH\nlog4j.appender.kafkaAppender.File=${kafka.logs.dir}/server.log\nlog4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout\nlog4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.appender.stateChangeAppender=org.apache.log4j.DailyRollingFileAppender\nlog4j.appender.stateChangeAppender.DatePattern='.'yyyy-MM-dd-HH\nlog4j.appender.stateChangeAppender.File=${kafka.logs.dir}/state-change.log\nlog4j.appender.stateChangeAppender.layout=org.apache.log4j.PatternLayout\nlog4j.appender.stateChangeAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.appender.requestAppender=org.apache.log4j.DailyRollingFileAppender\nlog4j.appender.requestAppender.DatePattern='.'yyyy-MM-dd-HH\nlog4j.appender.requestAppender.File=${kafka.logs.dir}/kafka-request.log\nlog4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout\nlog4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.appender.cleanerAppender=org.apache.log4j.DailyRollingFileAppender\nlog4j.appender.cleanerAppender.DatePattern='.'yyyy-MM-dd-HH\nlog4j.appender.cleanerAppender.File=${kafka.logs.dir}/log-cleaner.log\nlog4j.appender.cleanerAppender.layout=org.apache.log4j.PatternLayout\nlog4j.appender.cleanerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.appender.controllerAppender=org.apache.log4j.DailyRollingFileAppender\nlog4j.appender.controllerAppender.DatePattern='.'yyyy-MM-dd-HH\nlog4j.appender.controllerAppender.File=${kafka.logs.dir}/controller.log\nlog4j.appender.controllerAppender.layout=org.apache.log4j.PatternLayout\nlog4j.appender.controllerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.appender.authorizerAppender=org.apache.log4j.DailyRollingFileAppender\nlog4j.appender.authorizerAppender.DatePattern='.'yyyy-MM-dd-HH\nlog4j.appender.authorizerAppender.File=${kafka.logs.dir}/kafka-authorizer.log\nlog4j.appender.authorizerAppender.layout=org.apache.log4j.PatternLayout\nlog4j.appender.authorizerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.appender.ldapAppender=org.apache.log4j.DailyRollingFileAppender\nlog4j.appender.ldapAppender.DatePattern='.'yyyy-MM-dd-HH\nlog4j.appender.ldapAppender.File=${kafka.logs.dir}/kafka-ldap.log\nlog4j.appender.ldapAppender.layout=org.apache.log4j.PatternLayout\nlog4j.appender.ldapAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\n\n# Change the two lines below to adjust ZK client logging\nlog4j.logger.org.I0Itec.zkclient.ZkClient=INFO\nlog4j.logger.org.apache.zookeeper=INFO\n\n# Change the two lines below to adjust the general broker logging level (output to server.log and stdout)\nlog4j.logger.kafka=INFO\nlog4j.logger.org.apache.kafka=INFO\n\n# Change to DEBUG or TRACE to enable request logging\nlog4j.logger.kafka.request.logger=WARN, requestAppender\nlog4j.additivity.kafka.request.logger=false\n\n# Uncomment the lines below and change log4j.logger.kafka.network.RequestChannel$ to TRACE for additional output\n# related to the handling of requests\n#log4j.logger.kafka.network.Processor=TRACE, requestAppender\n#log4j.logger.kafka.server.KafkaApis=TRACE, requestAppender\n#log4j.additivity.kafka.server.KafkaApis=false\nlog4j.logger.kafka.network.RequestChannel$=WARN, requestAppender\nlog4j.additivity.kafka.network.RequestChannel$=false\n\nlog4j.logger.kafka.controller=TRACE, controllerAppender\nlog4j.additivity.kafka.controller=false\n\nlog4j.logger.kafka.log.LogCleaner=INFO, cleanerAppender\nlog4j.additivity.kafka.log.LogCleaner=false\n\nlog4j.logger.state.change.logger=TRACE, stateChangeAppender\nlog4j.additivity.state.change.logger=false\n\n# Access denials are logged at INFO level, change to DEBUG to also log allowed accesses\nlog4j.logger.kafka.authorizer.logger=DEBUG, authorizerAppender\nlog4j.additivity.kafka.authorizer.logger=false\n\n# Experimental, add logging for LDAP\nlog4j.logger.io.confluent.kafka.security.ldap.authorizer.LdapGroupManager=TRACE, ldapAppender\n\n"
},
{
"path": "ldap/kafka/server-with-ssl.properties",
"content": "# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# see kafka.server.KafkaConfig for additional details and defaults\n\n############################# Server Basics #############################\n\n# The id of the broker. This must be set to a unique integer for each broker.\nbroker.id=0\n\n############################# Socket Server Settings #############################\n\n# The address the socket server listens on. It will get the value returned from\n# java.net.InetAddress.getCanonicalHostName() if not configured.\n# FORMAT:\n# listeners = listener_name://host_name:port\n# EXAMPLE:\n# listeners = PLAINTEXT://your.host.name:9092\nlisteners=SASL_PLAINTEXT://kafka:9093\n\n# Hostname and port the broker will advertise to producers and consumers. If not set,\n# it uses the value for \"listeners\" if configured. Otherwise, it will use the value\n# returned from java.net.InetAddress.getCanonicalHostName().\nadvertised.listeners=SASL_PLAINTEXT://kafka:9093\n\n# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details\n#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL\n\nsecurity.inter.broker.protocol=SASL_PLAINTEXT\n\n# The number of threads that the server uses for receiving requests from the network and sending responses to the network\nnum.network.threads=3\n\n# The number of threads that the server uses for processing requests, which may include disk I/O\nnum.io.threads=8\n\n# The send buffer (SO_SNDBUF) used by the socket server\nsocket.send.buffer.bytes=102400\n\n# The receive buffer (SO_RCVBUF) used by the socket server\nsocket.receive.buffer.bytes=102400\n\n# The maximum size of a request that the socket server will accept (protection against OOM)\nsocket.request.max.bytes=104857600\n\n\n############################# Log Basics #############################\n\n# A comma separated list of directories under which to store log files\nlog.dirs=/var/lib/kafka\n\n# The default number of log partitions per topic. More partitions allow greater\n# parallelism for consumption, but this will also result in more files across\n# the brokers.\nnum.partitions=1\n\n# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.\n# This value is recommended to be increased for installations with data dirs located in RAID array.\nnum.recovery.threads.per.data.dir=1\n\n############################# Internal Topic Settings #############################\n# The replication factor for the group metadata internal topics \"__consumer_offsets\" and \"__transaction_state\"\n# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.\noffsets.topic.replication.factor=1\ntransaction.state.log.replication.factor=1\ntransaction.state.log.min.isr=1\n\n############################# Log Flush Policy #############################\n\n# Messages are immediately written to the filesystem but by default we only fsync() to sync\n# the OS cache lazily. The following configurations control the flush of data to disk.\n# There are a few important trade-offs here:\n# 1. Durability: Unflushed data may be lost if you are not using replication.\n# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.\n# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.\n# The settings below allow one to configure the flush policy to flush data after a period of time or\n# every N messages (or both). This can be done globally and overridden on a per-topic basis.\n\n# The number of messages to accept before forcing a flush of data to disk\n#log.flush.interval.messages=10000\n\n# The maximum amount of time a message can sit in a log before we force a flush\n#log.flush.interval.ms=1000\n\n############################# Log Retention Policy #############################\n\n# The following configurations control the disposal of log segments. The policy can\n# be set to delete segments after a period of time, or after a given size has accumulated.\n# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens\n# from the end of the log.\n\n# The minimum age of a log file to be eligible for deletion due to age\nlog.retention.hours=168\n\n# A size-based retention policy for logs. Segments are pruned from the log unless the remaining\n# segments drop below log.retention.bytes. Functions independently of log.retention.hours.\n#log.retention.bytes=1073741824\n\n# The maximum size of a log segment file. When this size is reached a new log segment will be created.\nlog.segment.bytes=1073741824\n\n# The interval at which log segments are checked to see if they can be deleted according\n# to the retention policies\nlog.retention.check.interval.ms=300000\n\n############################# Zookeeper #############################\n\n# Zookeeper connection string (see zookeeper docs for details).\n# This is a comma separated host:port pairs, each corresponding to a zk\n# server. e.g. \"127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002\".\n# You can also append an optional chroot string to the urls to specify the\n# root directory for all kafka znodes.\nzookeeper.connect=zookeeper:2181\n\n# Timeout in ms for connecting to zookeeper\nzookeeper.connection.timeout.ms=6000\n\n##################### Confluent Metrics Reporter #######################\n# Confluent Control Center and Confluent Auto Data Balancer integration\n#\n# Uncomment the following lines to publish monitoring data for\n# Confluent Control Center and Confluent Auto Data Balancer\n# If you are using a dedicated metrics cluster, also adjust the settings\n# to point to your metrics kakfa cluster.\n#metric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter\n#confluent.metrics.reporter.bootstrap.servers=localhost:9092\n#\n# Uncomment the following line if the metrics cluster has a single broker\n#confluent.metrics.reporter.topic.replicas=1\n\n##################### Confluent Proactive Support ######################\n# If set to true, and confluent-support-metrics package is installed\n# then the feature to collect and report support metrics\n# (\"Metrics\") is enabled. If set to false, the feature is disabled.\n#\nconfluent.support.metrics.enable=false\n\n\n# The customer ID under which support metrics will be collected and\n# reported.\n#\n# When the customer ID is set to \"anonymous\" (the default), then only a\n# reduced set of metrics is being collected and reported.\n#\n# Confluent customers\n# -------------------\n# If you are a Confluent customer, then you should replace the default\n# value with your actual Confluent customer ID. Doing so will ensure\n# that additional support metrics will be collected and reported.\n#\nconfluent.support.customer.id=anonymous\n\n############################# Group Coordinator Settings #############################\n\n# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.\n# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.\n# The default value for this is 3 seconds.\n# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.\n# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.\ngroup.initial.rebalance.delay.ms=0\n\n\n# SASL Configuration\nsasl.enabled.mechanisms=SCRAM-SHA-256\nsasl.mechanism.inter.broker.protocol=SCRAM-SHA-256\nsecurity.inter.broker.protocol=SASL_PLAINTEXT\nallow.everyone.if.no.acl.found=false\nsuper.users=User:kafka\nauthorizer.class.name=kafka.security.auth.SimpleAclAuthorizer\n\n# Configure authorizer\nauthorizer.class.name=io.confluent.kafka.security.ldap.authorizer.LdapAuthorizer\n# LDAP provider URL\nldap.authorizer.java.naming.provider.url=ldaps://ldap:636/DC=CONFLUENT,DC=IO\n# Refresh interval for LDAP cache. If set to zero, persistent search is used.\nldap.authorizer.refresh.interval.ms=60000\n\n# Lets see if we can connect with TLS to our LDAP server\nldap.authorizer.java.naming.security.principal=cn=admin,dc=confluent,dc=io\nldap.authorizer.java.naming.security.credentials=admin\n\nldap.authorizer.java.naming.security.protocol=SSL\nldap.authorizer.ssl.keystore.location=/etc/kafka/jks/ldap.keystore.jks\nldap.authorizer.ssl.keystore.password=confluent\n\nldap.authorizer.ssl.truststore.location=/etc/kafka/jks/ldap.truststore.jks\nldap.authorizer.ssl.truststore.password=confluent\n\n# Search base for group-based search\n#ldap.authorizer.group.search.base=ou=groups,dc=confluent,dc=io\n\n# Remember that LDAP works in a context. The search base is ou=groups,dc=confluent,dc=io\n# But since my URL is ldap://ldap:389/DC=CONFLUENT,DC=IO, we are already working in the dc=confluent,dc=io context\nldap.authorizer.group.search.base=ou=groups\n\n# Object class for groups\nldap.authorizer.group.object.class=posixGroup\nldap.authorizer.group.search.scope=2\n# Name of the attribute from which group name used in ACLs is obtained\nldap.authorizer.group.name.attribute=cn\n# Regex pattern to obtain group name used in ACLs from the attribute `ldap.authorizer.group.name.attribute`\nldap.authorizer.group.name.attribute.pattern=\n# Name of the attribute from which group members (user principals) are obtained\nldap.authorizer.group.member.attribute=memberUid\n# Regex pattern to obtain user principal from group member attribute\nldap.authorizer.group.member.attribute.pattern=cn=(.*),ou=users,dc=confluent,dc=io\n"
},
{
"path": "ldap/kafka/server.properties",
"content": "broker.id=0\nlisteners=SASL_PLAINTEXT://kafka:9093\nadvertised.listeners=SASL_PLAINTEXT://kafka:9093\nlog.dirs=/var/lib/kafka\noffsets.topic.replication.factor=1\ndefault.replication.factor=1\nconfluent.license.topic.replication.factor=1\ntransaction.state.log.replication.factor=1\ntransaction.state.log.min.isr=1\nzookeeper.connect=zookeeper:2181\n\n# SASL Configuration\nsecurity.inter.broker.protocol=SASL_PLAINTEXT\nsasl.enabled.mechanisms=SCRAM-SHA-256\nsasl.mechanism.inter.broker.protocol=SCRAM-SHA-256\nsecurity.inter.broker.protocol=SASL_PLAINTEXT\nallow.everyone.if.no.acl.found=false\nsuper.users=User:kafka\nauthorizer.class.name=kafka.security.auth.SimpleAclAuthorizer\n\n# Configure authorizer\nauthorizer.class.name=io.confluent.kafka.security.ldap.authorizer.LdapAuthorizer\n# LDAP provider URL\nldap.authorizer.java.naming.provider.url=ldap://ldap:389/DC=CONFLUENT,DC=IO\n# Refresh interval for LDAP cache. If set to zero, persistent search is used.\n# Reduced this value from the default 60000ms (60sec) to 10sec to detect\n# faster the updates done in the LDAP database\nldap.authorizer.refresh.interval.ms=10000\n\n# Lets try to see if we can run without security\nldap.authorizer.java.naming.security.authentication=SIMPLE\nldap.authorizer.java.naming.security.principal=cn=admin,dc=confluent,dc=io\nldap.authorizer.java.naming.security.credentials=admin\n\n# Search base for group-based search\n#ldap.authorizer.group.search.base=ou=groups,dc=confluent,dc=io\n\n# Remember that LDAP works in a context. The search base is ou=groups,dc=confluent,dc=io\n# But since my URL is ldap://ldap:389/DC=CONFLUENT,DC=IO, we are already working in the dc=confluent,dc=io context\nldap.authorizer.group.search.base=ou=groups\n\n# Object class for groups\nldap.authorizer.group.object.class=posixGroup\nldap.authorizer.group.search.scope=2\n# Name of the attribute from which group name used in ACLs is obtained\nldap.authorizer.group.name.attribute=cn\n# Regex pattern to obtain group name used in ACLs from the attribute `ldap.authorizer.group.name.attribute`\nldap.authorizer.group.name.attribute.pattern=\n# Name of the attribute from which group members (user principals) are obtained\nldap.authorizer.group.member.attribute=memberUid\n# Regex pattern to obtain user principal from group member attribute\nldap.authorizer.group.member.attribute.pattern=cn=(.*),ou=users,dc=confluent,dc=io\n"
},
{
"path": "ldap/kafka/users/purbon.properties",
"content": "sasl.mechanism=SCRAM-SHA-256\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"purbon\" \\\n password=\"purbon-secret\";\n"
},
{
"path": "ldap/ldap/certs/.gitignore",
"content": "*.crt\n*.csr\n*_creds\n*.jks\n*.srl\n*.key\n*.pem\n*.der\n*.p12\n"
},
{
"path": "ldap/ldap/custom/01_base.ldif",
"content": "dn: ou=users,{{ LDAP_BASE_DN }}\nobjectClass: organizationalUnit\nou: Users\n\ndn: ou=groups,{{ LDAP_BASE_DN }}\nobjectClass: organizationalUnit\nou: Groups\n\n"
},
{
"path": "ldap/ldap/custom/02_KafkaDevelopers.ldif",
"content": "dn: cn=Kafka Developers,ou=groups,{{ LDAP_BASE_DN }}\nobjectClass: top\nobjectClass: posixGroup\ncn: Kafka Developers\ngidNumber: 5000\n\n"
},
{
"path": "ldap/ldap/custom/10_alice.ldif",
"content": "dn: cn=alice,ou=users,{{ LDAP_BASE_DN }}\nobjectClass: inetOrgPerson\nobjectClass: posixAccount\nobjectClass: shadowAccount\nuid: alice\nsn: LookingGlass\ngivenName: Alice\ncn: alice\ndisplayName: Alice LookingGlass\nuidNumber: 10000\ngidNumber: 5000\nuserPassword: alice-secret\ngecos: alice\nloginShell: /bin/bash\nhomeDirectory: /home/alice\n"
},
{
"path": "ldap/ldap/custom/11_barnie.ldif",
"content": "dn: cn=barnie,ou=users,{{ LDAP_BASE_DN }}\nobjectClass: inetOrgPerson\nobjectClass: posixAccount\nobjectClass: shadowAccount\nuid: barnie\nsn: Rubble\ngivenName: Barnie\ncn: barnie\ndisplayName: Barnie Rubble\nuidNumber: 10001\ngidNumber: 5000\nuserPassword: barnie-secret\ngecos: barnie\nloginShell: /bin/bash\nhomeDirectory: /home/barnie\n"
},
{
"path": "ldap/ldap/custom/12_charlie.ldif",
"content": "dn: cn=charlie,ou=users,{{ LDAP_BASE_DN }}\nobjectClass: inetOrgPerson\nobjectClass: posixAccount\nobjectClass: shadowAccount\nuid: charlie\nsn: Sheen\ngivenName: Charlie\ncn: charlie\ndisplayName: Charlie Sheen\nuidNumber: 10001\ngidNumber: 5000\nuserPassword: charlie-secret\ngecos: charlie\nloginShell: /bin/bash\nhomeDirectory: /home/charlie\n"
},
{
"path": "ldap/ldap/custom/20_group_add.ldif",
"content": "dn: cn=Kafka Developers,ou=groups,{{ LDAP_BASE_DN }}\nchangetype: modify\nadd: memberuid\nmemberuid: cn=alice,ou=users,{{ LDAP_BASE_DN }}\n\ndn: cn=Kafka Developers,ou=groups,{{ LDAP_BASE_DN }}\nchangetype: modify\nadd: memberuid\nmemberuid: cn=barnie,ou=users,{{ LDAP_BASE_DN }}\n"
},
{
"path": "ldap/scripts/.gitignore",
"content": "*.crt\n*.csr\n*_creds\n*.jks\n*.srl\n*.key\n*.pem\n*.der\n*.p12\n"
},
{
"path": "ldap/scripts/certs-create.sh",
"content": "#!/bin/bash\n\n#set -o nounset \\\n# -o errexit \\\n# -o verbose \\\n# -o xtrace\n\n# Cleanup files\nrm -f *.crt *.csr *_creds *.jks *.srl *.key *.pem *.der *.p12\n\n# Generate CA key\nopenssl req -new -x509 -keyout snakeoil-ca-1.key -out snakeoil-ca-1.crt -days 365 -subj '/CN=ca1.test.confluent.io/OU=TEST/O=CONFLUENT/L=PaloAlto/S=Ca/C=US' -passin pass:confluent -passout pass:confluent\n\nfor i in kafka ldap\ndo\n\techo \"------------------------------- $i -------------------------------\"\n\n\t# Create host keystore\n\tkeytool -genkey -noprompt \\\n\t\t\t\t -alias $i \\\n\t\t\t\t -dname \"CN=$i,OU=TEST,O=CONFLUENT,L=PaloAlto,S=Ca,C=US\" \\\n -ext \"SAN=dns:$i,dns:localhost\" \\\n\t\t\t\t -keystore kafka.$i.keystore.jks \\\n\t\t\t\t -keyalg RSA \\\n\t\t\t\t -storepass confluent \\\n\t\t\t\t -keypass confluent\n\n\t# Create the certificate signing request (CSR)\n\tkeytool -keystore kafka.$i.keystore.jks -alias $i -certreq -file $i.csr -storepass confluent -keypass confluent -ext \"SAN=dns:$i,dns:localhost\"\n #openssl req -in $i.csr -text -noout\n\n # Sign the host certificate with the certificate authority (CA)\n openssl x509 -req -CA snakeoil-ca-1.crt -CAkey snakeoil-ca-1.key -in $i.csr -out $i-ca1-signed.crt -days 9999 -CAcreateserial -passin pass:confluent -extensions v3_req -extfile <(cat < ${i}_sslkey_creds\n \techo \"confluent\" > ${i}_keystore_creds\n \techo \"confluent\" > ${i}_truststore_creds\n\n\t# Create pem files and keys used for Schema Registry HTTPS testing\n\t# openssl x509 -noout -modulus -in client.certificate.pem | openssl md5\n\t# openssl rsa -noout -modulus -in client.key | openssl md5\n # echo \"GET /\" | openssl s_client -connect localhost:8085/subjects -cert client.certificate.pem -key client.key -tls1\n\tkeytool -export -alias $i -file $i.der -keystore kafka.$i.keystore.jks -storepass confluent\n\topenssl x509 -inform der -in $i.der -out $i.certificate.pem\n\tkeytool -importkeystore -srckeystore kafka.$i.keystore.jks -destkeystore $i.keystore.p12 -deststoretype PKCS12 -deststorepass confluent -srcstorepass confluent -noprompt\n\topenssl pkcs12 -in $i.keystore.p12 -nodes -nocerts -out $i.key -passin pass:confluent\n\ndone\n"
},
{
"path": "ldap/up",
"content": "#!/bin/sh\n\nusage() { echo \"Usage: $0 [--ssl] \" 1>&2; exit 1; }\n\nssl=0\nwhile getopts \":s-:\" opt; do\n case $opt in\n -)\n case \"${OPTARG}\" in\n ssl)\n ssl=1\n ;;\n *)\n usage\n exit 1\n ;;\n esac;;\n *)\n usage\n exit 1\n ;;\n esac\ndone\n\n## Select to run with security or not\n\nDOCKER_COMPOSE_FILE=\"$PWD/docker-compose.yaml\"\n\nif [ $ssl -eq 1 ]; then\n echo \"Running with SSL enabled between the brokers and the LDAP server\"\n # Generate the certificates\n cd scripts\n ./certs-create.sh\n\n ## Copy the necessary broker JKS stores\n cp kafka.kafka.keystore.jks ../kafka/jks/ldap.keystore.jks\n cp kafka.kafka.truststore.jks ../kafka/jks/ldap.truststore.jks\n\n ## copy the LDAP server certificates\n cp ldap-ca1-signed.crt ../ldap/certs/my-ldap.crt\n cp ldap.key ../ldap/certs/my-ldap.key\n cp snakeoil-ca-1.crt ../ldap/certs/my-ca.crt\n cd ..\n DOCKER_COMPOSE_FILE=\"$PWD/docker-compose-with-ssl.yaml\"\nfi\n\n## start docker-compose up to and including kafka\ndocker-compose -f $DOCKER_COMPOSE_FILE up -d --build kafka\n\n# Creating the users\n# kafka is configured as a super user\ndocker-compose exec kafka kafka-configs --bootstrap-server kafka:9093 --alter --add-config 'SCRAM-SHA-256=[password=kafka],SCRAM-SHA-512=[password=kafka]' --entity-type users --entity-name kafka\ndocker-compose exec kafka kafka-configs --zookeeper zookeeper:2181 --alter --add-config 'SCRAM-SHA-256=[password=alice-secret],SCRAM-SHA-512=[password=alice-secret]' --entity-type users --entity-name alice\ndocker-compose exec kafka kafka-configs --zookeeper zookeeper:2181 --alter --add-config 'SCRAM-SHA-256=[password=barnie-secret],SCRAM-SHA-512=[password=barnie-secret]' --entity-type users --entity-name barnie\ndocker-compose exec kafka kafka-configs --zookeeper zookeeper:2181 --alter --add-config 'SCRAM-SHA-256=[password=charlie-secret],SCRAM-SHA-512=[password=charlie-secret]' --entity-type users --entity-name charlie\n\ndocker-compose up -d\n\necho \"Example configuration:\"\necho \"Should succeed (barnie is in group)\"\necho \"-> docker-compose exec kafka kafka-console-producer --broker-list kafka:9093 --topic test-topic --producer.config=/etc/kafka/barnie.properties\"\necho \"Should fail (charlie is NOT in group)\"\necho \"-> docker-compose exec kafka kafka-console-producer --broker-list kafka:9093 --topic test-topic --producer.config=/etc/kafka/charlie.properties\"\necho \"Should succeed (alice is in group)\"\necho \"-> docker-compose exec kafka kafka-console-consumer --bootstrap-server kafka:9093 --consumer.config /etc/kafka/alice.properties --topic test-topic --from-beginning\"\necho \"List ACLs\"\necho \"-> docker-compose exec kafka kafka-acls --bootstrap-server kafka:9093 --list --command-config /etc/kafka/kafka.properties\"\n"
},
{
"path": "ldap/zookeeper/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER seknop@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.5/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-11-openjdk\nRUN yum install -y confluent-platform-2.12\n\n# 3. Configure Kafka and zookeeper for Kerberos \nCOPY zookeeper.properties /etc/kafka/zookeeper.properties\nCOPY zookeeper.sasl.jaas.config /etc/kafka/zookeeper_server_jaas.conf\n\nEXPOSE 2181\n\nCMD zookeeper-server-start /etc/kafka/zookeeper.properties \n"
},
{
"path": "ldap/zookeeper/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.5/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.5/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.5\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.5/archive.key\nenabled=1\n\n"
},
{
"path": "ldap/zookeeper/zookeeper.properties",
"content": "dataDir=/var/lib/zookeeper\nclientPort=2181\nmaxClientCnxns=0\nauthProvider.1 = org.apache.zookeeper.server.auth.SASLAuthenticationProvider\nrequireClientAuthScheme=sasl\n"
},
{
"path": "ldap/zookeeper/zookeeper.sasl.jaas.config",
"content": "Server {\n org.apache.zookeeper.server.auth.DigestLoginModule required\n user_kafka=\"kafka\";\n};\n"
},
{
"path": "ldap-auth/docker-compose.yaml",
"content": "version: '3'\nservices:\n ldap:\n image: osixia/openldap:1.3.0\n hostname: ldap\n container_name: ldap\n environment:\n LDAP_ORGANISATION: \"Confluent\"\n LDAP_DOMAIN: \"confluent.io\"\n ports:\n - \"389:389\"\n - \"636:636\"\n volumes:\n - \"$PWD/ldap/custom:/container/service/slapd/assets/config/bootstrap/ldif/custom\"\n command: \"--copy-service\"\n\n phpldapadmin-service:\n image: osixia/phpldapadmin:0.9.0\n container_name: ldapadmin-service\n environment:\n - PHPLDAPADMIN_LDAP_HOSTS=ldap\n ports:\n - \"6444:443\"\n depends_on:\n - ldap\n\n zookeeper:\n build: zookeeper/\n hostname: zookeeper\n container_name: zookeeper\n environment:\n - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/zookeeper_server_jaas.conf\n\n kafka:\n build: kafka/\n container_name: kafka\n environment:\n - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf\n depends_on:\n - zookeeper\n - ldap\n volumes:\n - \"$PWD/kafka/users:/service/kafka/users\"\n - \"$PWD/kafka/jks:/etc/kafka/jks\"\n ports:\n - \"9093:9093\"\n\n"
},
{
"path": "ldap-auth/kafka/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER seknop@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.5/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-11-openjdk\nRUN yum install -y confluent-server\nRUN yum install -y confluent-security\n\n# 3. Configure Kafka and zookeeper for Kerberos\nCOPY server.properties /etc/kafka/server.properties\nCOPY kafka.jaas.config /etc/kafka/kafka_server_jaas.conf\nCOPY log4j.properties /etc/kafka/log4j.properties\n\nCOPY alice.properties /etc/kafka/alice.properties\nCOPY barnie.properties /etc/kafka/barnie.properties\nCOPY charlie.properties /etc/kafka/charlie.properties\nCOPY kafka.properties /etc/kafka/kafka.properties\n\nEXPOSE 9093\n\nCMD kafka-server-start /etc/kafka/server.properties\n"
},
{
"path": "ldap-auth/kafka/alice.properties",
"content": "sasl.mechanism=PLAIN\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \\\n username=\"alice\" password=\"alice-secret\";\n"
},
{
"path": "ldap-auth/kafka/barnie.properties",
"content": "sasl.mechanism=PLAIN\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \\\n username=\"barnie\" \\\n password=\"barnie-secret\";\n"
},
{
"path": "ldap-auth/kafka/charlie.properties",
"content": "sasl.mechanism=PLAIN\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \\\n username=\"charlie\" \\\n password=\"charlie-secret\";\n"
},
{
"path": "ldap-auth/kafka/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.5/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.5/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.5\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.5/archive.key\nenabled=1\n\n"
},
{
"path": "ldap-auth/kafka/kafka.jaas.config",
"content": "Client {\n org.apache.zookeeper.server.auth.DigestLoginModule required\n username=\"kafka\"\n password=\"kafka\";\n};\n"
},
{
"path": "ldap-auth/kafka/kafka.properties",
"content": "sasl.mechanism=PLAIN\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \\\n username=\"kafka\" \\\n password=\"kafka\";\n"
},
{
"path": "ldap-auth/kafka/log4j.properties",
"content": "# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# Unspecified loggers and loggers with additivity=true output to server.log and stdout\n# Note that INFO only applies to unspecified loggers, the log level of the child logger is used otherwise\n# Sven is here!\nlog4j.rootLogger=INFO, stdout, kafkaAppender\n\nlog4j.appender.stdout=org.apache.log4j.ConsoleAppender\nlog4j.appender.stdout.layout=org.apache.log4j.PatternLayout\nlog4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender\nlog4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH\nlog4j.appender.kafkaAppender.File=${kafka.logs.dir}/server.log\nlog4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout\nlog4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.appender.stateChangeAppender=org.apache.log4j.DailyRollingFileAppender\nlog4j.appender.stateChangeAppender.DatePattern='.'yyyy-MM-dd-HH\nlog4j.appender.stateChangeAppender.File=${kafka.logs.dir}/state-change.log\nlog4j.appender.stateChangeAppender.layout=org.apache.log4j.PatternLayout\nlog4j.appender.stateChangeAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.appender.requestAppender=org.apache.log4j.DailyRollingFileAppender\nlog4j.appender.requestAppender.DatePattern='.'yyyy-MM-dd-HH\nlog4j.appender.requestAppender.File=${kafka.logs.dir}/kafka-request.log\nlog4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout\nlog4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.appender.cleanerAppender=org.apache.log4j.DailyRollingFileAppender\nlog4j.appender.cleanerAppender.DatePattern='.'yyyy-MM-dd-HH\nlog4j.appender.cleanerAppender.File=${kafka.logs.dir}/log-cleaner.log\nlog4j.appender.cleanerAppender.layout=org.apache.log4j.PatternLayout\nlog4j.appender.cleanerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.appender.controllerAppender=org.apache.log4j.DailyRollingFileAppender\nlog4j.appender.controllerAppender.DatePattern='.'yyyy-MM-dd-HH\nlog4j.appender.controllerAppender.File=${kafka.logs.dir}/controller.log\nlog4j.appender.controllerAppender.layout=org.apache.log4j.PatternLayout\nlog4j.appender.controllerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.appender.authorizerAppender=org.apache.log4j.DailyRollingFileAppender\nlog4j.appender.authorizerAppender.DatePattern='.'yyyy-MM-dd-HH\nlog4j.appender.authorizerAppender.File=${kafka.logs.dir}/kafka-authorizer.log\nlog4j.appender.authorizerAppender.layout=org.apache.log4j.PatternLayout\nlog4j.appender.authorizerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\n\nlog4j.appender.ldapAppender=org.apache.log4j.DailyRollingFileAppender\nlog4j.appender.ldapAppender.DatePattern='.'yyyy-MM-dd-HH\nlog4j.appender.ldapAppender.File=${kafka.logs.dir}/kafka-ldap.log\nlog4j.appender.ldapAppender.layout=org.apache.log4j.PatternLayout\nlog4j.appender.ldapAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\n\n# Change the two lines below to adjust ZK client logging\nlog4j.logger.org.I0Itec.zkclient.ZkClient=INFO\nlog4j.logger.org.apache.zookeeper=INFO\n\n# Change the two lines below to adjust the general broker logging level (output to server.log and stdout)\nlog4j.logger.kafka=INFO\nlog4j.logger.org.apache.kafka=INFO\n\n# Change to DEBUG or TRACE to enable request logging\nlog4j.logger.kafka.request.logger=WARN, requestAppender\nlog4j.additivity.kafka.request.logger=false\n\n# Uncomment the lines below and change log4j.logger.kafka.network.RequestChannel$ to TRACE for additional output\n# related to the handling of requests\n#log4j.logger.kafka.network.Processor=TRACE, requestAppender\n#log4j.logger.kafka.server.KafkaApis=TRACE, requestAppender\n#log4j.additivity.kafka.server.KafkaApis=false\nlog4j.logger.kafka.network.RequestChannel$=WARN, requestAppender\nlog4j.additivity.kafka.network.RequestChannel$=false\n\nlog4j.logger.kafka.controller=TRACE, controllerAppender\nlog4j.additivity.kafka.controller=false\n\nlog4j.logger.kafka.log.LogCleaner=INFO, cleanerAppender\nlog4j.additivity.kafka.log.LogCleaner=false\n\nlog4j.logger.state.change.logger=TRACE, stateChangeAppender\nlog4j.additivity.state.change.logger=false\n\n# Access denials are logged at INFO level, change to DEBUG to also log allowed accesses\nlog4j.logger.kafka.authorizer.logger=DEBUG, authorizerAppender\nlog4j.additivity.kafka.authorizer.logger=false\n\n# Experimental, add logging for LDAP\nlog4j.logger.io.confluent.kafka.security.ldap.authorizer.LdapGroupManager=TRACE, ldapAppender\n\n"
},
{
"path": "ldap-auth/kafka/server.properties",
"content": "broker.id=0\nlisteners=SASL_PLAINTEXT://kafka:9093\nadvertised.listeners=SASL_PLAINTEXT://kafka:9093\nlog.dirs=/var/lib/kafka\noffsets.topic.replication.factor=1\ntransaction.state.log.replication.factor=1\ntransaction.state.log.min.isr=1\nzookeeper.connect=zookeeper:2181\n\ninter.broker.listener.name=SASL_PLAINTEXT\nsasl.mechanism.inter.broker.protocol=PLAIN\n\n# SASL Configuration\nlistener.name.sasl_plaintext.sasl.enabled.mechanisms=PLAIN\nlistener.name.sasl_plaintext.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafka\" password=\"kafka\";\n\nlistener.name.sasl_plaintext.plain.sasl.server.callback.handler.class=io.confluent.security.auth.provider.ldap.LdapAuthenticateCallbackHandler\n\n# LDAP authentication\nldap.java.naming.provider.url=ldap://ldap:389\nldap.java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory\n# Authenticate to LDAP\nldap.java.naming.security.principal=cn=admin,dc=confluent,dc=io\nldap.java.naming.security.credentials=admin\nldap.java.naming.security.authentication=SIMPLE\n# Locate users\nldap.user.search.base=ou=users,dc=confluent,dc=io\nldap.user.name.attribute=uid\nldap.user.object.class=inetOrgPerson\n\nconfluent.support.metrics.enable=false\nconfluent.license.topic.replication.factor=1\n\n# ldap.user.password.attribute=userPassword\n\n# allow.everyone.if.no.acl.found=false\n# super.users=User:kafka\n#authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer\n#sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \\\n# username=\"kafka\" \\\n# password=\"kafka\" ;\n\n# Configure authorizer\n# authorizer.class.name=io.confluent.kafka.security.ldap.authorizer.LdapAuthorizer\n# authorizer.class.name=io.confluent.kafka.security.authorizer.ConfluentServerAuthorizer\n\n# LDAP provider URL\n# ldap.authorizer.java.naming.provider.url=ldap://ldap:389\n# Refresh interval for LDAP cache. If set to zero, persistent search is used.\n# Reduced this value from the default 60000ms (60sec) to 10sec to detect\n# faster the updates done in the LDAP database\n# ldap.authorizer.refresh.interval.ms=10000\n\n# ldap.authorizer.java.naming.security.authentication=SIMPLE\n# ldap.authorizer.java.naming.security.principal=cn=admin,dc=confluent,dc=io\n# ldap.authorizer.java.naming.security.credentials=admin\n\n# Search base for group-based search\n#ldap.authorizer.group.search.base=ou=groups,dc=confluent,dc=io\n\n# Remember that LDAP works in a context. The search base is ou=groups,dc=confluent,dc=io\n# But since my URL is ldap://ldap:389/DC=CONFLUENT,DC=IO, we are already working in the dc=confluent,dc=io context\n# ldap.authorizer.group.search.base=ou=groups,dc=confluent,dc=io\n\n# Object class for groups\n# ldap.authorizer.group.object.class=posixGroup\n#ldap.authorizer.group.search.scope=2\n# Name of the attribute from which group name used in ACLs is obtained\n# ldap.authorizer.group.name.attribute=cn\n# Regex pattern to obtain group name used in ACLs from the attribute `ldap.authorizer.group.name.attribute`\n# ldap.authorizer.group.name.attribute.pattern=\n# Name of the attribute from which group members (user principals) are obtained\n# ldap.authorizer.group.member.attribute=memberUid\n# Regex pattern to obtain user principal from group member attribute\n# ldap.authorizer.group.member.attribute.pattern=cn=(.*),ou=users,dc=confluent,dc=io\n"
},
{
"path": "ldap-auth/ldap/custom/01_base.ldif",
"content": "dn: ou=users,dc=confluent,dc=io\nobjectClass: organizationalUnit\nou: Users\n\ndn: ou=groups,dc=confluent,dc=io\nobjectClass: organizationalUnit\nou: Groups\n\n"
},
{
"path": "ldap-auth/ldap/custom/02_KafkaDevelopers.ldif",
"content": "dn: cn=Kafka Developers,ou=groups,{{ LDAP_BASE_DN }}\nobjectClass: top\nobjectClass: posixGroup\ncn: Kafka Developers\ngidNumber: 5000\n\n"
},
{
"path": "ldap-auth/ldap/custom/03_ProjectA.ldif",
"content": "dn: cn=ProjectA,ou=groups,{{ LDAP_BASE_DN }}\nobjectClass: top\nobjectClass: posixGroup\ncn: ProjectA\ngidNumber: 5001\n\n"
},
{
"path": "ldap-auth/ldap/custom/04_ProjectB.ldif",
"content": "dn: cn=ProjectB,ou=groups,{{ LDAP_BASE_DN }}\nobjectClass: top\nobjectClass: posixGroup\ncn: ProjectB\ngidNumber: 5002\n\n"
},
{
"path": "ldap-auth/ldap/custom/10_alice.ldif",
"content": "dn: cn=alice,ou=users,{{ LDAP_BASE_DN }}\nobjectClass: inetOrgPerson\nobjectClass: posixAccount\nobjectClass: shadowAccount\nuid: alice\nsn: LookingGlass\ngivenName: Alice\ncn: alice\ndisplayName: Alice LookingGlass\nuidNumber: 10000\ngidNumber: 5000\nuserPassword: alice-secret\ngecos: alice\nloginShell: /bin/bash\nhomeDirectory: /home/alice\n"
},
{
"path": "ldap-auth/ldap/custom/11_barnie.ldif",
"content": "dn: cn=barnie,ou=users,{{ LDAP_BASE_DN }}\nobjectClass: inetOrgPerson\nobjectClass: posixAccount\nobjectClass: shadowAccount\nuid: barnie\nsn: Rubble\ngivenName: Barnie\ncn: barnie\ndisplayName: Barnie Rubble\nuidNumber: 10001\ngidNumber: 5000\nuserPassword: barnie-secret\ngecos: barnie\nloginShell: /bin/bash\nhomeDirectory: /home/barnie\n"
},
{
"path": "ldap-auth/ldap/custom/12_charlie.ldif",
"content": "dn: cn=charlie,ou=users,{{ LDAP_BASE_DN }}\nobjectClass: inetOrgPerson\nobjectClass: posixAccount\nobjectClass: shadowAccount\nuid: charlie\nsn: Sheen\ngivenName: Charlie\ncn: charlie\ndisplayName: Charlie Sheen\nuidNumber: 10002\ngidNumber: 5000\nuserPassword: charlie-secret\ngecos: charlie\nloginShell: /bin/bash\nhomeDirectory: /home/charlie\n"
},
{
"path": "ldap-auth/ldap/custom/13_donald.ldif",
"content": "dn: cn=donald,ou=users,{{ LDAP_BASE_DN }}\nobjectClass: inetOrgPerson\nobjectClass: posixAccount\nobjectClass: shadowAccount\nuid: donald\nsn: Duck\ngivenName: Donald\ncn: donald\ndisplayName: Donald Duck\nuidNumber: 10003\ngidNumber: 5000\nuserPassword: donald-secret\ngecos: donald\nloginShell: /bin/bash\nhomeDirectory: /home/donald\n"
},
{
"path": "ldap-auth/ldap/custom/14_eva.ldif",
"content": "dn: cn=eva,ou=users,{{ LDAP_BASE_DN }}\nobjectClass: inetOrgPerson\nobjectClass: posixAccount\nobjectClass: shadowAccount\nuid: eva\nsn: Maria\ngivenName: Eva\ncn: eva\ndisplayName: Eva Maria\nuidNumber: 10004\ngidNumber: 5000\nuserPassword: eva-secret\ngecos: eva\nloginShell: /bin/bash\nhomeDirectory: /home/eva\n"
},
{
"path": "ldap-auth/ldap/custom/15_fritz.ldif",
"content": "dn: cn=fritz,ou=users,{{ LDAP_BASE_DN }}\nobjectClass: inetOrgPerson\nobjectClass: posixAccount\nobjectClass: shadowAccount\nuid: fritz\nsn: Walter\ngivenName: Fritz\ncn: fritz\ndisplayName: Fritz Walter\nuidNumber: 10005\ngidNumber: 5000\nuserPassword: fritz-secret\ngecos: fritz\nloginShell: /bin/bash\nhomeDirectory: /home/fritz\n"
},
{
"path": "ldap-auth/ldap/custom/16_greta.ldif",
"content": "dn: cn=greta,ou=users,{{ LDAP_BASE_DN }}\nobjectClass: inetOrgPerson\nobjectClass: posixAccount\nobjectClass: shadowAccount\nuid: greta\nsn: Thunberg\ngivenName: Greta\ncn: greta\ndisplayName: Greta Thunberg\nuidNumber: 10006\ngidNumber: 5000\nuserPassword: greta-secret\ngecos: greta\nloginShell: /bin/bash\nhomeDirectory: /home/greta\n"
},
{
"path": "ldap-auth/ldap/custom/17_kafka.ldif",
"content": "dn: cn=kafka,ou=users,{{ LDAP_BASE_DN }}\nobjectClass: inetOrgPerson\nobjectClass: posixAccount\nobjectClass: shadowAccount\nuid: kafka\nsn: kafka\ngivenName: kafka\ncn: kafka\ndisplayName: kafka\nuidNumber: 10007\ngidNumber: 5000\nuserPassword: kafka\ngecos: kafka\nloginShell: /bin/bash\nhomeDirectory: /home/kafka\n"
},
{
"path": "ldap-auth/ldap/custom/20_group_add.ldif",
"content": "dn: cn=Kafka Developers,ou=groups,{{ LDAP_BASE_DN }}\nchangetype: modify\nadd: memberuid\nmemberuid: cn=alice,ou=users,{{ LDAP_BASE_DN }}\n\ndn: cn=Kafka Developers,ou=groups,{{ LDAP_BASE_DN }}\nchangetype: modify\nadd: memberuid\nmemberuid: cn=barnie,ou=users,{{ LDAP_BASE_DN }}\n\ndn: cn=Kafka Developers,ou=groups,{{ LDAP_BASE_DN }}\nchangetype: modify\nadd: memberuid\nmemberuid: cn=charlie,ou=users,{{ LDAP_BASE_DN }}\n\ndn: cn=Kafka Developers,ou=groups,{{ LDAP_BASE_DN }}\nchangetype: modify\nadd: memberuid\nmemberuid: cn=eva,ou=users,{{ LDAP_BASE_DN }}\n\ndn: cn=Kafka Developers,ou=groups,{{ LDAP_BASE_DN }}\nchangetype: modify\nadd: memberuid\nmemberuid: cn=fritz,ou=users,{{ LDAP_BASE_DN }}\n"
},
{
"path": "ldap-auth/up",
"content": "#!/bin/sh\n\n## start docker-compose up to and including kafka\ndocker-compose up -d --build \n\necho \"Example configuration:\"\necho \"Should succeed (barnie is in group)\"\necho \"-> docker-compose exec kafka kafka-console-producer --broker-list kafka:9093 --topic test-topic --producer.config=/etc/kafka/barnie.properties\"\necho \"Should fail (charlie is NOT in group)\"\necho \"-> docker-compose exec kafka kafka-console-producer --broker-list kafka:9093 --topic test-topic --producer.config=/etc/kafka/charlie.properties\"\necho \"Should succeed (alice is in group)\"\necho \"-> docker-compose exec kafka kafka-console-consumer --bootstrap-server kafka:9093 --consumer.config /etc/kafka/alice.properties --topic test-topic --from-beginning\"\necho \"List ACLs\"\necho \"-> docker-compose exec kafka kafka-acls --bootstrap-server kafka:9093 --list --command-config /etc/kafka/kafka.properties\"\n"
},
{
"path": "ldap-auth/zookeeper/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER seknop@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.5/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-11-openjdk\nRUN yum install -y confluent-platform-2.12\n\n# 3. Configure Kafka and zookeeper for Kerberos \nCOPY zookeeper.properties /etc/kafka/zookeeper.properties\nCOPY zookeeper.sasl.jaas.config /etc/kafka/zookeeper_server_jaas.conf\n\nEXPOSE 2181\n\nCMD zookeeper-server-start /etc/kafka/zookeeper.properties \n"
},
{
"path": "ldap-auth/zookeeper/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.5/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.5/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.5\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.5/archive.key\nenabled=1\n\n"
},
{
"path": "ldap-auth/zookeeper/zookeeper.properties",
"content": "dataDir=/var/lib/zookeeper\nclientPort=2181\nmaxClientCnxns=0\nauthProvider.1 = org.apache.zookeeper.server.auth.SASLAuthenticationProvider\nrequireClientAuthScheme=sasl\n"
},
{
"path": "ldap-auth/zookeeper/zookeeper.sasl.jaas.config",
"content": "Server {\n org.apache.zookeeper.server.auth.DigestLoginModule required\n user_kafka=\"kafka\";\n};\n"
},
{
"path": "multi-sasl/docker-compose.yml",
"content": "version: '3'\nservices:\n\n zookeeper:\n build: zookeeper/\n container_name: zookeeper\n hostname: zookeeper\n restart: on-failure\n environment:\n - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/zookeeper_server_jaas.conf \n\n kafka:\n build: kafka/\n container_name: kafka\n environment:\n - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf\n depends_on: \n - zookeeper\n restart: on-failure\n"
},
{
"path": "multi-sasl/kafka/Dockerfile",
"content": "FROM centos\nMAINTAINER seknop@gmail.com\nENV container docker\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.4/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-11-openjdk-devel\nRUN yum install -y confluent-platform-2.12\n\n# 3. Configure Kafka and zookeeper for Kerberos \nCOPY server.properties /etc/kafka/server.properties\nCOPY kafka.sasl.jaas.config /etc/kafka/kafka_server_jaas.conf\nCOPY consumer.properties /etc/kafka/consumer.properties\nCOPY consumer.plain.properties /etc/kafka/consumer.plain.properties\n\nEXPOSE 9093\n\nCMD kafka-server-start /etc/kafka/server.properties\n"
},
{
"path": "multi-sasl/kafka/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.4/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.4\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n"
},
{
"path": "multi-sasl/kafka/consumer.plain.properties",
"content": "sasl.mechanism=PLAIN\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \\\n username=\"kafka\" \\\n password=\"kafka\";\n\n"
},
{
"path": "multi-sasl/kafka/consumer.properties",
"content": "sasl.mechanism=SCRAM-SHA-256\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"kafka\" \\\n password=\"kafka\";\n\n"
},
{
"path": "multi-sasl/kafka/kafka.sasl.jaas.config",
"content": "Client {\n org.apache.zookeeper.server.auth.DigestLoginModule required\n username=\"admin\"\n password=\"password\";\n};\n"
},
{
"path": "multi-sasl/kafka/server.properties",
"content": "broker.id=0\nlisteners=SASL_PLAINTEXT://kafka:9093\nadvertised.listeners=SASL_PLAINTEXT://kafka:9093\nlog.dirs=/var/lib/kafka\noffsets.topic.replication.factor=1\ntransaction.state.log.replication.factor=1\ntransaction.state.log.min.isr=1\nzookeeper.connect=zookeeper:2181\n\n# Scram Authentication mechanism\nsasl.enabled.mechanisms=SCRAM-SHA-256,PLAIN\nsasl.mechanism.inter.broker.protocol=SCRAM-SHA-256\nlistener.name.sasl_plaintext.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \\\n username=\"kafka\" \\\n password=\"kafka\";\nlistener.name.sasl_plaintext.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \\\n username=\"kafka\" \\\n password=\"kafka\" \\\n user_kafka=\"kafka\";\nsecurity.inter.broker.protocol=SASL_PLAINTEXT\nallow.everyone.if.no.acl.found=false\nsuper.users=User:kafka\nauthorizer.class.name=kafka.security.auth.SimpleAclAuthorizer\nzookeeper.set.acl=true\n"
},
{
"path": "multi-sasl/up",
"content": "#!/bin/sh\n\ndocker-compose up -d --build\n\n# Creating the user kafka\n# kafka is configured as a super user, no need for additional ACL\ndocker-compose exec kafka kafka-configs --zookeeper zookeeper:2181 --alter --add-config 'SCRAM-SHA-256=[password=kafka],SCRAM-SHA-512=[password=kafka]' --entity-type users --entity-name kafka\n\necho \"Example configuration:\"\necho \"-> docker-compose exec kafka kafka-console-producer --broker-list kafka:9093 --producer.config /etc/kafka/consumer.properties --topic test\"\necho \"-> docker-compose exec kafka kafka-console-producer --broker-list kafka:9093 --producer.config /etc/kafka/consumer.plain.properties --topic test\"\necho \"-> docker-compose exec kafka kafka-console-consumer --bootstrap-server kafka:9093 --consumer.config /etc/kafka/consumer.properties --topic test --from-beginning\"\n"
},
{
"path": "multi-sasl/zookeeper/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.4/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-11-openjdk-devel\nRUN yum install -y confluent-platform-2.12\n\n# 3. Configure Kafka and zookeeper for Kerberos \nCOPY zookeeper.properties /etc/kafka/zookeeper.properties\nCOPY zookeeper.sasl.jaas.config /etc/kafka/zookeeper_server_jaas.conf\n\nEXPOSE 2181\n\nCMD zookeeper-server-start /etc/kafka/zookeeper.properties \n"
},
{
"path": "multi-sasl/zookeeper/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.4/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.4\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n"
},
{
"path": "multi-sasl/zookeeper/zookeeper.properties",
"content": "dataDir=/var/lib/zookeeper\nclientPort=2181\nmaxClientCnxns=0\nauthProvider.1 = org.apache.zookeeper.server.auth.SASLAuthenticationProvider\n"
},
{
"path": "multi-sasl/zookeeper/zookeeper.sasl.jaas.config",
"content": "Server {\n org.apache.zookeeper.server.auth.DigestLoginModule required\n user_admin=\"password\";\n};\n"
},
{
"path": "none/docker-compose.yml",
"content": "---\nversion: '3'\nservices:\n zookeeper:\n image: confluentinc/cp-zookeeper:6.1.0\n hostname: zookeeper\n container_name: zookeeper\n environment:\n ZOOKEEPER_SERVER_ID: 1\n ZOOKEEPER_CLIENT_PORT: 2181\n ZOOKEEPER_TICK_TIME: \"2000\"\n ZOOKEEPER_SERVERS: zookeeper:2888:3888\n KAFKA_JMX_PORT: 9999\n KAFKA_JMX_HOSTNAME: localhost\n kafka:\n image: confluentinc/cp-enterprise-kafka:6.1.0\n hostname: kafka\n container_name: kafka\n depends_on:\n - zookeeper\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_LISTENER: INTERNAL://kafka:9092,OUTSIDE://localhost:9093\n KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,OUTSIDE://localhost:9093\n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT\n KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL\n KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter\n CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: kafka:9092\n KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1\n KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0\n KAFKA_JMX_PORT: 9999\n KAFKA_JMX_HOSTNAME: kafka\n KAFKA_BROKER_RACK: 0\n ports:\n - 9093:9093\n \n\n\n\n"
},
{
"path": "none/up",
"content": "#!/bin/sh\n\ndocker-compose up -d\n\n# Creating the user kafka\n# kafka is configured as a super user, no need for additional ACL\n# docker-compose exec kafka kafka-configs --zookeeper zookeeper:2181 --alter --add-config 'SCRAM-SHA-255=[password=kafka],SCRAM-SHA-512=[password=kafka]' --entity-type users --entity-name kafka\n\necho \"Example configuration:\"\necho \"-> docker-compose exec kafka kafka-console-producer --broker-list kafka:9092 --topic test\"\necho \"-> docker-compose exec kafka kafka-console-consumer --bootstrap-server kafka:9092 --topic test --from-beginning\"\n"
},
{
"path": "oauth/.gitignore",
"content": "certs/\n*.jks\n"
},
{
"path": "oauth/ca.cnf",
"content": "[ policy_match ]\ncountryName = match\nstateOrProvinceName = match\norganizationName = match\norganizationalUnitName = optional\ncommonName = supplied\nemailAddress = optional\n\n[ req ]\nprompt = no\ndistinguished_name = dn\ndefault_md = sha256\ndefault_bits = 4096\nx509_extensions = v3_ca\n\n[ dn ]\ncountryName = UK\norganizationName = Confluent\nlocalityName = London\ncommonName = kafka.confluent.local\n\n[ v3_ca ]\nsubjectKeyIdentifier=hash\nbasicConstraints = critical,CA:true\nauthorityKeyIdentifier=keyid:always,issuer:always\nkeyUsage = critical,keyCertSign,cRLSign\n"
},
{
"path": "oauth/docker-compose.yml",
"content": "version: '3'\nservices:\n\n zookeeper:\n build: zookeeper/\n container_name: zookeeper\n domainname: confluent.local\n hostname: zookeeper\n networks:\n default:\n aliases:\n - zookeeper.confluent.local\n\n kafka:\n build: kafka/\n container_name: kafka\n domainname: confluent.local\n hostname: kafka\n depends_on: \n - zookeeper\n environment:\n - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf\n networks:\n default:\n aliases:\n - kafka.confluent.local\n\nnetworks:\n default:\n"
},
{
"path": "oauth/generate_certs.sh",
"content": "#!/bin/bash\n\nset -u\nset -e\n\nvalidity=1\nkeystore_pass=secret\nkey_pass=secret\ndistinguished_name=kafka.confluent.local\nhostname=kafka.confluent.local\n\nrm -rf certs\nmkdir certs\npushd certs\n\n# Generate server certificate\nkeytool -keystore kafka.server.keystore.jks \\\n -alias localhost \\\n -validity ${validity} \\\n -genkey -keyalg RSA \\\n -storepass ${keystore_pass} \\\n -key-pass ${key_pass} \\\n -dname \"CN=kafka.confluent.local, OU=CS, O=Confluent, L=Palo Alto, S=CA, C=US\" \\\n -ext SAN=DNS:${hostname}\n\n# keytool -list -v -keystore kafka.server.keystore.jks -storepass ${keystore_pass}\n\n# Generate certificate auithority\nopenssl req -new -nodes -x509 -newkey rsa:2048 -keyout ca-key -out ca-cert -days ${validity} -config ../ca.cnf\n\necho \"Importing certificate authority into client truststore.\"\nkeytool -keystore kafka.client.truststore.jks \\\n -alias CARoot \\\n -import \\\n -file ca-cert \\\n -store-pass ${keystore_pass} \\\n -no-prompt\n\necho \"Importing certificate authority into server truststore.\"\nkeytool -keystore kafka.server.truststore.jks \\\n -alias CARoot \\\n -import \\\n -file ca-cert \\\n -store-pass ${keystore_pass} \\\n -no-prompt\n\necho \"Exporting server certificate such that it can be signed.\"\nkeytool -keystore kafka.server.keystore.jks \\\n -alias localhost \\\n -certreq \\\n -file cert-file \\\n -store-pass ${keystore_pass} \n\necho \"Signing exported server certificate.\"\nopenssl x509 -req \\\n -CA ca-cert \\\n -CAkey ca-key \\\n -in cert-file \\\n -out cert-signed \\\n -days ${validity} \\\n -CAcreateserial \\\n -passin pass:{ca-password}\n\necho \"Importing certificate authority into keystore.\"\nkeytool -keystore kafka.server.keystore.jks \\\n -alias CARoot \\\n -import -file ca-cert \\\n -store-pass ${keystore_pass} \\\n -no-prompt\n \necho \"Importing signed certificate into server keystore.\"\nkeytool -keystore kafka.server.keystore.jks \\\n -alias localhost \\\n -import \\\n -file cert-signed \\\n -store-pass ${keystore_pass} \\\n -no-prompt\n \necho \"Moving client truststore and kafka truststore and kafka keystore to Kafka docker directory\"\npopd\nmv certs/kafka.server.truststore.jks kafka\nmv certs/kafka.server.keystore.jks kafka\nmv certs/kafka.client.truststore.jks kafka\n\n"
},
{
"path": "oauth/kafka/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER d.gasparina@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.4/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-1.8.0-openjdk\nRUN yum install -y confluent-platform-2.12\n\n# 3. Configure Kafka and zookeeper \nCOPY server.properties /etc/kafka/server.properties\nCOPY client.properties /etc/kafka/client.properties\nCOPY kafka_server_jaas.conf /etc/kafka/kafka_server_jaas.conf\nCOPY oauthcallbackhandlers/target/dummy-oauth-adapter-0.1.0-jar-with-dependencies.jar /usr/share/java/kafka/dummy-oauth-adapter-0.1.0-jar-with-dependencies.jar\nCOPY test_produce_and_consume.sh /tmp/test_produce_and_consume.sh\n\n# 4. Put SSL certificates in place\nCOPY kafka.server.keystore.jks /etc/kafka/kafka.server.keystore.jks\nCOPY kafka.server.truststore.jks /etc/kafka/kafka.server.truststore.jks\n# this will be used by the kafka-console-producer.sh and kafka-console-consumer.sh scripts\nCOPY kafka.client.truststore.jks /etc/kafka/kafka.client.truststore.jks\n\nEXPOSE 9093\n\nCMD kafka-server-start /etc/kafka/server.properties\n"
},
{
"path": "oauth/kafka/client.properties",
"content": "security.protocol=SASL_SSL\nsasl.mechanism=OAUTHBEARER\nsasl.login.callback.handler.class=io.confluent.examples.authentication.oauth.OauthBearerLoginCallbackHandler\nssl.truststore.location=/etc/kafka/kafka.client.truststore.jks\nssl.truststore.password=secret\n"
},
{
"path": "oauth/kafka/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.4/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.4\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n"
},
{
"path": "oauth/kafka/kafka_server_jaas.conf",
"content": "KafkaServer {\n org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;\n};\n\nKafkaClient {\n org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;\n};\n"
},
{
"path": "oauth/kafka/oauthcallbackhandlers/.gitignore",
"content": "target/\n.idea\n"
},
{
"path": "oauth/kafka/oauthcallbackhandlers/pom.xml",
"content": "\n\n 4.0.0\n io.confluent.cs.examples\n dummy-oauth-adapter\n 0.1.0\n\n \n 1.8\n 1.8\n \n\n \n \n org.apache.kafka\n kafka-clients\n 2.4.0\n \n \n junit\n junit\n 4.13.1\n test\n \n \n com.fasterxml.jackson.core\n jackson-databind\n 2.10.0.pr1\n \n \n \n jakarta.xml.bind\n jakarta.xml.bind-api\n 2.3.2\n \n\n \n \n org.glassfish.jaxb\n jaxb-runtime\n 2.3.2\n \n \n \n io.jsonwebtoken\n jjwt\n 0.9.1\n \n \n org.projectlombok\n lombok\n 1.18.0\n \n \n\n \n \n \n org.apache.maven.plugins\n maven-assembly-plugin\n 3.1.1\n \n \n jar-with-dependencies\n \n\n \n \n \n assemble-all\n package\n \n single\n \n \n \n \n \n \n\n"
},
{
"path": "oauth/kafka/oauthcallbackhandlers/src/main/java/io/confluent/examples/authentication/oauth/JwtHelper.java",
"content": "package io.confluent.examples.authentication.oauth;\n\nimport io.jsonwebtoken.Claims;\nimport io.jsonwebtoken.Jws;\nimport io.jsonwebtoken.Jwts;\nimport io.jsonwebtoken.SignatureAlgorithm;\n\nimport java.io.UnsupportedEncodingException;\nimport java.util.Arrays;\nimport java.util.Date;\nimport java.util.HashSet;\n\npublic class JwtHelper {\n\n String createJwt() throws UnsupportedEncodingException {\n return Jwts.builder()\n .setSubject(\"bene\")\n .setExpiration(new Date(System.currentTimeMillis() + 1000 * 60 * 60))\n .claim(\"name\", \"Benedikt\")\n .claim(\"scope\", \"developer admin\")\n .setNotBefore(new Date())\n .setIssuedAt(new Date())\n .signWith(\n SignatureAlgorithm.HS256,\n \"secret\".getBytes(\"UTF-8\")\n ).compact();\n }\n\n MyOauthBearerToken validate(String jwt) throws UnsupportedEncodingException {\n Jws claims = Jwts.parser()\n .setSigningKey(\"secret\".getBytes(\"UTF-8\"))\n .parseClaimsJws(jwt);\n MyOauthBearerToken token = new MyOauthBearerToken();\n token.setLifetimeMs(claims.getBody().getExpiration().getTime());\n token.setPrincipalName(claims.getBody().getSubject());\n token.setScopes(new HashSet<>(Arrays.asList(((String) claims.getBody().get(\"scope\")).split(\" \"))));\n token.setStartTimeMs(claims.getBody().getIssuedAt().getTime());\n token.setValue(jwt);\n return token;\n }\n\n}\n"
},
{
"path": "oauth/kafka/oauthcallbackhandlers/src/main/java/io/confluent/examples/authentication/oauth/MyOauthBearerToken.java",
"content": "package io.confluent.examples.authentication.oauth;\n\nimport lombok.Data;\nimport org.apache.kafka.common.security.oauthbearer.OAuthBearerToken;\n\nimport java.util.HashSet;\nimport java.util.Set;\n\n@Data\npublic class MyOauthBearerToken implements OAuthBearerToken {\n\n private long lifetimeMs;\n private String value;\n private long startTimeMs;\n private String principalName;\n private Set scopes = new HashSet<>();\n\n MyOauthBearerToken() { }\n\n MyOauthBearerToken(String value) {\n this.value = value;\n this.lifetimeMs = System.currentTimeMillis() + 1000 * 60 * 60;\n }\n\n @Override\n public String value() {\n return this.value;\n }\n\n @Override\n public Set scope() {\n return scopes;\n }\n\n @Override\n public long lifetimeMs() {\n return this.lifetimeMs;\n }\n\n @Override\n public String principalName() {\n return this.principalName;\n }\n\n @Override\n public Long startTimeMs() {\n return startTimeMs;\n }\n}\n"
},
{
"path": "oauth/kafka/oauthcallbackhandlers/src/main/java/io/confluent/examples/authentication/oauth/OauthBearerLoginCallbackHandler.java",
"content": "package io.confluent.examples.authentication.oauth;\n\nimport org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;\nimport org.apache.kafka.common.security.oauthbearer.OAuthBearerTokenCallback;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport javax.security.auth.callback.Callback;\nimport javax.security.auth.callback.UnsupportedCallbackException;\nimport javax.security.auth.login.AppConfigurationEntry;\nimport java.io.UnsupportedEncodingException;\nimport java.util.List;\nimport java.util.Map;\n\npublic class OauthBearerLoginCallbackHandler implements AuthenticateCallbackHandler {\n\n private final Logger log = LoggerFactory.getLogger(OauthBearerLoginCallbackHandler.class);\n\n private JwtHelper jwtHelper = new JwtHelper();\n\n @Override\n public void configure(Map configs, String saslMechanism, List jaasConfigEntries) {\n\n }\n\n @Override\n public void close() {\n\n }\n\n @Override\n public void handle(Callback[] callbacks) throws UnsupportedCallbackException, UnsupportedEncodingException {\n log.info(\"Handling callbacks.\");\n for (Callback callback : callbacks) {\n if (callback instanceof OAuthBearerTokenCallback) {\n OAuthBearerTokenCallback oAuthBearerTokenCallback = (OAuthBearerTokenCallback) callback;\n // TODO: a bearer token would usually be retrieved from an authorization server.\n oAuthBearerTokenCallback.token(new MyOauthBearerToken(jwtHelper.createJwt()));\n log.info(\"Created jwt compact form\");\n continue;\n }\n throw new UnsupportedCallbackException(callback);\n }\n\n }\n}\n\n\n"
},
{
"path": "oauth/kafka/oauthcallbackhandlers/src/main/java/io/confluent/examples/authentication/oauth/OauthBearerValidatorCallbackHandler.java",
"content": "package io.confluent.examples.authentication.oauth;\n\nimport org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;\nimport org.apache.kafka.common.security.oauthbearer.OAuthBearerValidatorCallback;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport javax.security.auth.callback.Callback;\nimport javax.security.auth.callback.UnsupportedCallbackException;\nimport javax.security.auth.login.AppConfigurationEntry;\nimport java.io.IOException;\nimport java.util.List;\nimport java.util.Map;\n\npublic class OauthBearerValidatorCallbackHandler implements AuthenticateCallbackHandler {\n\n private final Logger log = LoggerFactory.getLogger(OauthBearerValidatorCallbackHandler.class);\n\n private JwtHelper jwtHelper = new JwtHelper();\n\n @Override\n public void configure(Map configs, String saslMechanism, List jaasConfigEntries) {\n\n }\n\n @Override\n public void close() {\n\n }\n\n @Override\n public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {\n log.info(\"Validating token.\");\n for (Callback callback : callbacks) {\n if (callback instanceof OAuthBearerValidatorCallback) {\n OAuthBearerValidatorCallback oAuthBearerValidatorCallback = (OAuthBearerValidatorCallback) callback;\n log.info(\"Tokenvalue: {}\", oAuthBearerValidatorCallback.tokenValue());\n oAuthBearerValidatorCallback.token(jwtHelper.validate(oAuthBearerValidatorCallback.tokenValue()));\n continue;\n }\n throw new UnsupportedCallbackException(callback);\n }\n }\n}\n"
},
{
"path": "oauth/kafka/oauthcallbackhandlers/src/test/java/io/confluent/examples/authentication/oauth/JwtHelperTest.java",
"content": "package io.confluent.examples.authentication.oauth;\n\nimport org.junit.Test;\n\nimport java.io.UnsupportedEncodingException;\nimport java.util.Arrays;\nimport java.util.HashSet;\n\nimport static org.junit.Assert.assertEquals;\nimport static org.junit.Assert.assertTrue;\n\npublic class JwtHelperTest {\n\n @Test\n public void test() throws UnsupportedEncodingException {\n JwtHelper underTest = new JwtHelper();\n String jwt = underTest.createJwt();\n MyOauthBearerToken parsed = underTest.validate(jwt);\n System.err.println(parsed);\n assertEquals(\"bene\", parsed.getPrincipalName());\n assertEquals(new HashSet<>(Arrays.asList(\"developer\", \"admin\")), parsed.getScopes());\n assertTrue(parsed.getStartTimeMs() <= System.currentTimeMillis());\n assertTrue(parsed.getLifetimeMs() > System.currentTimeMillis());\n }\n}\n"
},
{
"path": "oauth/kafka/oauthcallbackhandlers/src/test/java/io/confluent/examples/authentication/oauth/ProduceDataTest.java",
"content": "package io.confluent.examples.authentication.oauth;\n\nimport org.apache.kafka.clients.producer.KafkaProducer;\nimport org.apache.kafka.clients.producer.ProducerConfig;\nimport org.apache.kafka.clients.producer.ProducerRecord;\nimport org.apache.kafka.clients.producer.RecordMetadata;\nimport org.apache.kafka.common.security.auth.SecurityProtocol;\nimport org.apache.kafka.common.serialization.StringSerializer;\nimport org.junit.Ignore;\nimport org.junit.Test;\n\nimport java.util.Properties;\nimport java.util.concurrent.ExecutionException;\nimport java.util.concurrent.Future;\n\npublic class ProduceDataTest {\n\n // This test will not work until advertised listeners is set correctly in docker-compose.yml .\n @Test\n @Ignore\n public void test() throws ExecutionException, InterruptedException {\n Properties producerConfig = new Properties();\n producerConfig.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, \"localhost:9093\");\n producerConfig.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);\n producerConfig.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);\n producerConfig.put(ProducerConfig.RETRIES_CONFIG, 0);\n producerConfig.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 3000);\n producerConfig.put(ProducerConfig.LINGER_MS_CONFIG, 1000);\n producerConfig.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 1000);\n producerConfig.put(\"sasl.mechanism\", \"OAUTHBEARER\");\n producerConfig.put(\"security.protocol\", SecurityProtocol.SASL_PLAINTEXT.name);\n producerConfig.put(\"sasl.jaas.config\", \"org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required unsecuredLoginStringClaim_sub=\\\"alice\\\";\");\n producerConfig.put(\"sasl.login.callback.handler.class\", \"io.confluent.examples.authentication.oauth.OauthBearerLoginCallbackHandler\");\n KafkaProducer kafkaProducer = new KafkaProducer<>(producerConfig);\n Future result = kafkaProducer.send(new ProducerRecord<>(\"foo\", \"bar\"));\n result.get();\n }\n}\n"
},
{
"path": "oauth/kafka/server.properties",
"content": "############################# Server Basics #############################\nbroker.id=0\nlisteners=SASL_SSL://kafka.confluent.local:9093\nadvertised.listeners=SASL_SSL://kafka.confluent.local:9093\nlog.dirs=/var/lib/kafka\noffsets.topic.replication.factor=1\ntransaction.state.log.replication.factor=1\ntransaction.state.log.min.isr=1\nzookeeper.connect=zookeeper.confluent.local:2181\n\n# oauth bearer configuration\nsecurity.inter.broker.protocol=SASL_SSL\nsasl.mechanism.inter.broker.protocol=OAUTHBEARER\nsasl.enabled.mechanisms=OAUTHBEARER\nlistener.name.sasl_ssl.oauthbearer.sasl.server.callback.handler.class=io.confluent.examples.authentication.oauth.OauthBearerValidatorCallbackHandler\nlistener.name.sasl_ssl.oauthbearer.sasl.login.callback.handler.class=io.confluent.examples.authentication.oauth.OauthBearerLoginCallbackHandler\n\nssl.truststore.location=/etc/kafka/kafka.server.truststore.jks\nssl.truststore.password=secret\nssl.keystore.location=/etc/kafka/kafka.server.keystore.jks\nssl.keystore.password=secret\nssl.key.password=secret\n"
},
{
"path": "oauth/kafka/test_produce_and_consume.sh",
"content": "#!/bin/bash \n\necho 'some sample messages\nsent via sasl outh bearer authentication\nwith custom token generation and validation. \n' | kafka-console-producer --broker-list kafka.confluent.local:9093 --topic test --producer.config /etc/kafka/client.properties\ntimeout 5 kafka-console-consumer --bootstrap-server kafka.confluent.local:9093 --topic test --from-beginning --consumer.config /etc/kafka/client.properties\n"
},
{
"path": "oauth/up",
"content": "#!/bin/sh\n\nset -e\nset -u\n\npushd kafka/oauthcallbackhandlers\nmvn clean package\npopd\n\n./generate_certs.sh\n\ndocker-compose up -d --build\n\nsleep 5\n\ndocker-compose exec kafka /tmp/test_produce_and_consume.sh\n"
},
{
"path": "oauth/zookeeper/Dockerfile",
"content": "FROM centos:centos8\nMAINTAINER seknop@gmail.com\nENV container docker\n\n# 0. Fixing Mirror list for Centos\nRUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*\nRUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.4/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install zookeeper and kafka\nRUN yum install -y java-1.8.0-openjdk\nRUN yum install -y confluent-platform-2.12\n\n# 3. Configure Kafka and zookeeper \nCOPY zookeeper.properties /etc/kafka/zookeeper.properties\n\nEXPOSE 2181\n\nCMD zookeeper-server-start /etc/kafka/zookeeper.properties \n"
},
{
"path": "oauth/zookeeper/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.4/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.4\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.4/archive.key\nenabled=1\n\n"
},
{
"path": "oauth/zookeeper/zookeeper.properties",
"content": "dataDir=/var/lib/zookeeper\nclientPort=2181\nmaxClientCnxns=0\n"
},
{
"path": "plain/consumer.properties",
"content": "sasl.mechanism=PLAIN\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \\\n username=\"consumer\" \\\n password=\"consumer-secret\";\n\n"
},
{
"path": "plain/docker-compose.yml",
"content": "version: '3.4'\nservices:\n\n zookeeper:\n image: ${REPOSITORY}/cp-zookeeper:${TAG}\n container_name: zookeeper\n hostname: zookeeper\n environment:\n ZOOKEEPER_CLIENT_PORT: 2181\n ZOOKEEPER_TICK_TIME: 2000\n\n kafka:\n image: ${REPOSITORY}/cp-kafka:${TAG}\n container_name: kafka\n hostname: kafka\n depends_on:\n - zookeeper\n ports:\n - 9093:9093\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'\n KAFKA_LISTENER: INTERNAL://kafka:9092,OUTSIDE://localhost:9093\n KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,OUTSIDE://localhost:9093\n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:SASL_PLAINTEXT,OUTSIDE:SASL_PLAINTEXT\n KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL\n KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN\n KAFKA_LISTENER_NAME_INTERNAL_SASL_ENABLED_MECHANISMS: PLAIN\n KAFKA_LISTENER_NAME_OUTSIDE_SASL_ENABLED_MECHANISMS: PLAIN\n\n KAFKA_LISTENER_NAME_INTERNAL_PLAIN_SASL_JAAS_CONFIG: org.apache.kafka.common.security.plain.PlainLoginModule required \\\n username=\"admin\" \\\n password=\"admin-secret\" \\\n user_admin=\"admin-secret\" ;\n KAFKA_LISTENER_NAME_OUTSIDE_PLAIN_SASL_JAAS_CONFIG: org.apache.kafka.common.security.plain.PlainLoginModule required \\\n username=\"admin\" \\\n password=\"admin-secret\" \\\n user_admin=\"admin-secret\" \\\n user_producer=\"producer-secret\" \\\n user_consumer=\"consumer-secret\" ;\n KAFKA_SASL_JAAS_CONFIG: org.apache.kafka.common.security.plain.PlainLoginModule required \\\n username=\"admin\" \\\n password=\"admin-secret\" ;\n KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1\n"
},
{
"path": "plain/producer.properties",
"content": "sasl.mechanism=PLAIN\nsecurity.protocol=SASL_PLAINTEXT\nsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \\\n username=\"producer\" \\\n password=\"producer-secret\";\n\n"
},
{
"path": "plain/up",
"content": "#!/bin/sh\n\ndocker-compose up -d\n\necho \"Example configuration:\"\necho \"-> kafka-console-producer --broker-list localhost:9093 --producer.config producer.properties --topic test\"\necho \"-> kafka-console-consumer --bootstrap-server localhost:9093 --consumer.config consumer.properties --topic test --from-beginning\"\n"
},
{
"path": "quotas/Client/Dockerfile",
"content": "FROM centos\nMAINTAINER seknop@gmail.com\nENV container docker\n\n# 1. Adding Confluent repository\nRUN rpm --import https://packages.confluent.io/rpm/5.1/archive.key\nCOPY confluent.repo /etc/yum.repos.d/confluent.repo\nRUN yum clean all\n\n# 2. Install Confluent Kafka stack\nRUN yum install -y java-1.8.0-openjdk\nRUN yum install -y confluent-kafka-2.11\n\nCMD tail -f /dev/null\n"
},
{
"path": "quotas/Client/confluent.repo",
"content": "[Confluent.dist]\nname=Confluent repository (dist)\nbaseurl=https://packages.confluent.io/rpm/5.1/7\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.1/archive.key\nenabled=1\n\n[Confluent]\nname=Confluent repository\nbaseurl=https://packages.confluent.io/rpm/5.1\ngpgcheck=1\ngpgkey=https://packages.confluent.io/rpm/5.1/archive.key\nenabled=1\n"
},
{
"path": "quotas/Grafana/provisioning/dashboards/grafana-dashboard.json",
"content": "{\n \"__inputs\": [\n {\n \"name\": \"DS_PROMETHEUS\",\n \"label\": \"Prometheus\",\n \"description\": \"\",\n \"type\": \"datasource\",\n \"pluginId\": \"prometheus\",\n \"pluginName\": \"Prometheus\"\n }\n ],\n \"__requires\": [\n {\n \"type\": \"grafana\",\n \"id\": \"grafana\",\n \"name\": \"Grafana\",\n \"version\": \"6.1.1\"\n },\n {\n \"type\": \"panel\",\n \"id\": \"graph\",\n \"name\": \"Graph\",\n \"version\": \"\"\n },\n {\n \"type\": \"datasource\",\n \"id\": \"prometheus\",\n \"name\": \"Prometheus\",\n \"version\": \"1.0.0\"\n }\n ],\n \"annotations\": {\n \"list\": [\n {\n \"builtIn\": 1,\n \"datasource\": \"-- Grafana --\",\n \"enable\": true,\n \"hide\": true,\n \"iconColor\": \"rgba(0, 211, 255, 1)\",\n \"name\": \"Annotations & Alerts\",\n \"type\": \"dashboard\"\n }\n ]\n },\n \"editable\": true,\n \"gnetId\": null,\n \"graphTooltip\": 0,\n \"id\": null,\n \"links\": [],\n \"panels\": [\n {\n \"aliasColors\": {},\n \"bars\": false,\n \"dashLength\": 10,\n \"dashes\": false,\n \"datasource\": \"Prometheus\",\n \"fill\": 1,\n \"gridPos\": {\n \"h\": 9,\n \"w\": 12,\n \"x\": 0,\n \"y\": 0\n },\n \"id\": 2,\n \"legend\": {\n \"avg\": false,\n \"current\": false,\n \"max\": false,\n \"min\": false,\n \"show\": true,\n \"total\": false,\n \"values\": false\n },\n \"lines\": true,\n \"linewidth\": 1,\n \"nullPointMode\": \"null\",\n \"percentage\": false,\n \"pointradius\": 2,\n \"points\": false,\n \"renderer\": \"flot\",\n \"seriesOverrides\": [],\n \"spaceLength\": 10,\n \"stack\": false,\n \"steppedLine\": false,\n \"targets\": [\n {\n \"expr\": \"cp_kafka_server_throttle_produce_quota_throttle\",\n \"format\": \"time_series\",\n \"hide\": false,\n \"intervalFactor\": 1,\n \"refId\": \"A\"\n }\n ],\n \"thresholds\": [],\n \"timeFrom\": null,\n \"timeRegions\": [],\n \"timeShift\": null,\n \"title\": \"Panel Title\",\n \"tooltip\": {\n \"shared\": true,\n \"sort\": 0,\n \"value_type\": \"individual\"\n },\n \"type\": \"graph\",\n \"xaxis\": {\n \"buckets\": null,\n \"mode\": \"time\",\n \"name\": null,\n \"show\": true,\n \"values\": []\n },\n \"yaxes\": [\n {\n \"format\": \"short\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": null,\n \"min\": null,\n \"show\": true\n },\n {\n \"format\": \"short\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": null,\n \"min\": null,\n \"show\": true\n }\n ],\n \"yaxis\": {\n \"align\": false,\n \"alignLevel\": null\n }\n }\n ],\n \"refresh\": \"5s\",\n \"schemaVersion\": 18,\n \"style\": \"dark\",\n \"tags\": [],\n \"templating\": {\n \"list\": []\n },\n \"time\": {\n \"from\": \"now-5m\",\n \"to\": \"now\"\n },\n \"timepicker\": {\n \"refresh_intervals\": [\n \"5s\",\n \"10s\",\n \"30s\",\n \"1m\",\n \"5m\",\n \"15m\",\n \"30m\",\n \"1h\",\n \"2h\",\n \"1d\"\n ],\n \"time_options\": [\n \"5m\",\n \"15m\",\n \"1h\",\n \"6h\",\n \"12h\",\n \"24h\",\n \"2d\",\n \"7d\",\n \"30d\"\n ]\n },\n \"timezone\": \"\",\n \"title\": \"throttle-one-user\",\n \"uid\": \"M4aaf_RWz\",\n \"version\": 1\n}\n"
},
{
"path": "quotas/Grafana/provisioning/dashboards/one-quota.yaml",
"content": "apiVersion: 1\n\nproviders:\n # provider name\n- name: 'prometheus'\n # org id. will default to orgId 1 if not specified\n orgId: 1\n # name of the dashboard folder. Required\n folder: ''\n # folder UID. will be automatically generated if not specified\n folderUid: ''\n # provider type. Required\n type: file\n # disable dashboard deletion\n disableDeletion: false\n # enable dashboard editing\n editable: true\n # how often Grafana will scan for changed dashboards\n updateIntervalSeconds: 10 \n options:\n # path to dashboard files on disk. Required\n path: /etc/grafana/provisioning/dashboards\n"
},
{
"path": "quotas/Grafana/provisioning/datasources/prometheus.yaml",
"content": "# config file version\napiVersion: 1\n\n# list of datasources that should be deleted from the database\ndeleteDatasources:\n - name: Prometheus\n orgId: 1\n\n# list of datasources to insert/update depending\n# whats available in the database\ndatasources:\n # name of the datasource. Required\n- name: Prometheus\n # datasource type. Required\n type: prometheus\n # access mode. direct or proxy. Required\n access: proxy\n # org id. will default to orgId 1 if not specified\n orgId: 1\n # url\n url: http://prometheus:9090\n # database password, if used\n password:\n # database user, if used\n user:\n # database name, if used\n database:\n # enable/disable basic auth\n basicAuth: true\n # basic auth username\n basicAuthUser: admin\n # basic auth password\n basicAuthPassword: foobar\n # enable/disable with credentials headers\n withCredentials:\n # mark as default datasource. Max one per org\n isDefault: true\n #