Kafka Connect 2.6 gives source connectors a topic-creation policy
Source connectors can create topics with configured partition and replication settings through an administrative client. This separates connector provisioning from broker-wide automatic topic creation.
A source connector can discover data that needs a new Kafka topic, but broker-wide automatic creation gives that topic the broker's general policy. Kafka 2.6 adds a more specific place to describe the required topic settings. The 3 August release lets configured source connectors create their topics through an administrative client. That connects topic provisioning to the connector that will produce the records.
The implementation makes this separate from the broker's automatic-creation feature. The released WorkerConfig says each task uses an admin client and does not depend on brokers automatically creating topics. At the worker level, the topic.creation.enable option defaults to true for this new administrative creation mechanism, but the connector must also supply topic.creation properties for the feature to apply.
I regard the partition key as a central design decision in Kafka because ordering applies within a partition. Topic partition count and replication settings are likewise consequential configuration. Letting the connector express them can be useful, provided that convenience does not obscure which policy a newly discovered topic receives.
The worker permits creation and the connector describes it
The released SourceConnectorConfig activates enriched topic-creation configuration when creation is permitted and the supplied properties contain the topic-creation prefix. This is a two-level decision. A worker can disallow the feature, while an enabled worker can execute connector-specific creation settings.
That scope matters when interpreting the default. The worker option being true does not mean every source connector acquires an arbitrary topic policy without configuration. It permits a configured connector to use the mechanism. An operator can separate whether Connect may manage topics from what one connector requests for the topics it creates.
The configuration also supports named groups. topic.creation.groups provides aliases for sets of topic-creation properties. Each group can describe which topic names it includes or excludes and which settings should apply. This makes the policy useful for a source that produces more than one kind of topic, without requiring every topic to use a single uniform declaration.
The default group must specify the core settings
TopicCreationConfig requires a partition count and replication factor for the default group. Each can be positive, or minus one to request the broker's default. Other groups inherit those values from the default group when they omit them. The implementation validates the basic ranges before using the configuration.
The following is a connector-configuration fragment, not a complete connector definition. It specifies three partitions and a replication factor of three for topics using its default creation policy. The target cluster must have enough brokers to satisfy that replication factor.
topic.creation.default.partitions=3
topic.creation.default.replication.factor=3
The distinction between an explicit value and minus one is useful. A connector can state a concrete requirement or deliberately defer to a broker default. Both choices are visible in the configuration. Silently relying on a broker-wide creation mechanism would leave that choice outside the connector's description.
The reference says requesting a replication factor greater than the number of brokers causes topic creation to fail. A syntactically valid positive value is therefore not sufficient evidence that provisioning can succeed. The request still has to be meaningful for the target cluster. Connect supplies the policy, while the cluster must be able to realize it.
Matching rules are part of provisioning
Named groups use regular-expression lists for topic inclusion and exclusion. The configuration description gives exclusion precedence over inclusion. The source connector configuration also removes include and exclude expressions supplied for the default group, so those expressions do not narrow its fallback behavior. This keeps named matching rules separate from the baseline settings.
Those details make topic names operational inputs. A pattern determines which policy a discovered topic receives. A naming change can therefore affect more than readability if it changes which group matches. The settings should be interpreted together with the actual topic naming behavior of the connector that produces them.
I find this arrangement useful because it keeps discovery and provisioning related without making them identical. A source can discover a new stream, while the configured groups determine the Kafka resources requested for it. The topic-creation policy does not decide what the external records mean or select the application's partition key.
The release also changes default connection behavior
The 2.6 upgrade notes enable TLS 1.3 by default on Java 11 or newer. Client and server negotiate it when both support it and otherwise fall back to TLS 1.2. That is a separate step from the earlier addition of TLS 1.3 support. Runtime version and peer support remain conditions for the negotiated result.
The same notes change client.dns.lookup to use_all_dns_ips. When a hostname resolves to several addresses, clients and brokers try the addresses in sequence until a connection succeeds. This changes the default interpretation of a multi-address hostname. It does not require an application to replace its bootstrap hostname with a hard-coded list of addresses.
These connection changes matter to a Connect worker because its administrative and data paths both depend on reaching Kafka. Topic creation adds an explicit administrative action before records can use the new topic. DNS and TLS settings determine how those connections are established, while the connector's creation groups determine what resources it requests after connecting.
The practical addition in Kafka 2.6 is a source-specific provisioning contract. The worker decides whether the mechanism is permitted, and the connector supplies a default policy with optional named groups. New topics can consequently receive settings chosen for that integration through an explicit create operation, even when the broker's general automatic-creation behavior is not the mechanism providing them.