20.6 | Kurrent Docs
Skip to main content

20.6

EventStoreDB 20.6

  • Released: 09 June 2020
  • Supported until: October 2020

20.6.1

28 September 2020

In this release

Insecure mode

We have introduced an insecure mode for those who wish to run EventStoreDB without security or certificates.

Insecure mode disables all authentication, authorization, and transport security for the node and any clients connected to it.

This option can be enabled with the --insecure flag.

Custom client certificate common name

The client certificate common name (CN) that is used to secure a cluster was previously hardcoded to eventstoredb-node. This meant that public CAs could not be used to generate a certificate for the cluster.

The client certificate CN can now be set with the CertificateReservedNodeCommonName option.

Reloading certificates

Certificates can be reloaded by triggering the /admin/reloadconfig endpoint, or by sending a SIGHUP signal on Linux systems.

The process for rolling the certificates on a cluster is as follows:

  1. Update the certificates on each node one by one. Update the config file if required (e.g., if certificate file names, password, or paths have changed)
  2. Send a POST request to the /admin/reloadconfig endpoint, or a SIGHUP signal to the EventStoreDB process.

Reloading log levels

It's sometimes useful to update the log levels to be more verbose while debugging issues with EventStoreDB. Previously, this required a restart of the EventStoreD which could cause the issue to stop presenting itself.

An endpoint has been added to allow reloading the log level from configuration while EventStoreD is running.

To make use of this, update the LogLevel option in the config file, and then send a POST request to the /admin/reloadconfig endpoint, or send a SIGHUP signal if on Linux.

Restart persistent subscriptions

Persistent subscriptions can sometimes run into issues that require them to be restarted. Previously, you'd need to restart the leader node to fix the issue, which is disruptive to the cluster.

The persistent subscriptions service can now be restarted without restarting the leader node. This can be done either by sending a POST request to the /subscriptions/restart HTTP endpoint, or through the persistent subscriptions gRPC client.

Breaking changes

Dev mode removed

Dev mode didn't make sense following the introduction of a dedicated insecure mode.

Due to this and the fact that dev mode should not have been used in production, we made the decision to remove dev mode entirely.

If you were using dev mode to avoid setting up certificates, please run EventStoreDB in insecure mode instead. If you are wanting to run EventStoreDB with certificates, please refer to the documentation.

Default cluster gossip port

The cluster gossip port was previously set to 30777 by default as this was the default gossip port used by Manager nodes.

Since EventStoreDB 20.6 does not use manager nodes, keeping the port set to this value made no sense. As such, the default cluster gossip port is now the default HTTP port for a node, 2113.

Embedded TCP client

The Embedded TCP client is not a part of the 20.6.1 release. There are some known issues with the client in this version, specifically around running in a clustered setup.

20.6.0

09 June 2020

In this release

Development mode

Currently the easiest way to run EventStoreDB is to run in Development mode. You can do this by running EventStoreDB with the --dev flag.

This configures EventStoreDB to:

  • Use an existing development certificate. This means that when running in development mode, you don't need to specify a certificate at startup.
  • Run in memory. By default, development mode will not write any data to disk. You can change this behaviour by overriding the MemDb setting.

Configuring a cluster to run with Certificates

If you would like to run EventStoreDB without development mode, you will need to generate your own certificates. There are some requirements for these, especially if you intend on running a cluster of nodes.

These requirements are:

  • At least one IP address or DNS name needs to be specified in the SAN. The specified IP addresses or DNS names must match the IP/host of the EventStoreDB node.
  • The certificate's subject needs to be CN=eventstoredb-node.
  • You will also need to configure EventStoreDB with the path to the CA used to create these certificates.

Once you have the certificates generated, you will need to configure your nodes to use them. You can do this by adding the following configuration to any existing configuration:

TrustedRootCertificatesPath: {Path to CA Directory}
CertificateFile: {Path to Certificate File}
CertificatePrivateKeyFile: {Path to Private Key File} # Optional
CertificatePassword: changeit # Optional

You will need to either trust the certificate authority you are running EventStoreDB with, or disable remote certificate validation on your client.

Breaking changes

Cluster role name change

The terminology used for the cluster node roles has been changed from Master and Slave to Leader and Follower respectively.

Following this change, there are a few things to take into consideration:

  • The ES-RequireMaster header for HTTP will be deprecated in a future release and ES-RequireLeader should be used going forwards
  • Monitoring scripts that leverage the gossip protocol will need to be updated to the new terminology

HTTP prefixes removed

HTTP prefixes were dropped with the move from HTTPListener to Kestrel. Please be aware that EventStoreDB will fail if you try to specify HTTP Prefixes in the configuration.

HTTP interface now uses HTTPS

The external HTTP interface has been changed to use HTTPS only. This means that a TLS certificate must be provided at startup to secure the interface.

This also means that external gossip is now over HTTPS. This may affect any cluster health status monitoring you have in place, as well as TCP Clients connecting to the cluster.

The TCP client has been updated to gossip over HTTPS by default. If you are using an older client, you will need to manually change it to use HTTPS when gossiping.

TCP client has TLS enabled by default

With EventStoreDB 20.6.0 we want to move to a secure by default approach and therefore the TCP client has been changed to have TLS Enabled by default.

Removal of undocumented projection selectors

Some projection selectors and options were removed:

  • fromStreamCatalog selector
  • fromStreamsMatching selector
  • disableParallelism option
  • catalogTransform option

These selectors were only usable in queries or transient projections. If you were using them, you need to recreate any affected queries or transient projections.

As only fromStreamsMatching was officially documented, it was the only one of these officially supported. Usage of fromStreamsMatching can be replaced with fromAll with an appropriate where modifier.

Mutually exclusive security on TCP

Each TCP interface can be started as either secure or non secured, but not both at the same time. This can be controlled with the --disable-internal-tls and --disable-external-tls options. TLS is enabled for both interfaces by default.

Combined internal and external HTTP interfaces

We have combined the internal and external HTTP interfaces and therefore the previous configuration options of ExtHttpPort and IntHttpPort has been combined into HttpPort and IntHttpPortAdvertiseAs and ExtHttpPortAdvertiseAs into HttpPortAdvertiseAs.

Logging infrastructure moved to Serilog

The logging system has been moved over to Serilog. This does mean that the previous logging configuration is no longer valid for users who are upgrading.

At this time, the only configurable options in the provided logconfig.json are the log levels for various logging outputs.

Deprecated features

Deprecation of Atom and TCP

As we move over to gRPC we are deprecating AtomPub and TCP. Both of these are disabled by default and command line options have been added to the server to enable both TCP and AtomPub.

  • AtomPub over HTTP can be enabled with --enable-atom-pub-over-http.
  • The external TCP interface can be enabled with --enable-external-tcp.

Note: AtomPub is enabled with dev mode, as being able to browse streams is a useful development tool.

Deprecation of clone nodes

Following the introduction of Read-Only Replicas, Clones have been disabled by default.

The new default behaviour is for excess nodes in the cluster to be terminated upon joining. If you need to make use of Clone nodes while transitioning to using Read-Only Replicas, the old behaviour can be restored by setting the --unsafe-allow-surplus-nodes option on all nodes in the cluster.

Main headline features

Important notes for macOS and the use of Docker

Although macOS was never a supported server platform for EventStoreDB, we have historically shipped binaries for those developing on that platform. In this preview release we have completed the move to gRPC for both client access and gossip which enables TLS by default.

The gRPC implementation for .NET Core uses the system-native TLS library for the platform on which it is running. In the case of macOS server-side ALPN is not supported by Security.Framework and consequently the Core CLR. As a result of this, we are no longer shipping binaries for macOS server until such time as this is rectified. We recommend that users on macOS use the Docker container build of the server instead.

The gRPC client will still be supported on Linux, Windows and macOS.

.NET Core 3.1

EventStoreDB 20.6.0 only runs on Core CLR 3.1 which means that support for both the .NET Framework and Mono have been dropped.

Much of the platform-specific code has been removed, and we benefit from the focus on performance that Microsoft has given the Core CLR. We now use the Kestrel HTTP server, which has significant benefits over HttpListener which was used up to version 5.

Historically we have provided Linux and macOS binaries with the Mono runtime statically linked into them. Unfortunately this is not supported by .NET Core. Consequently we now ship the correct version of the runtime as part of our packages.

Introduction of gRPC

Since version 1.0 of EventStoreDB, two client protocols have been supported - Atompub over HTTP, and full-duplex protocol-buffers-over-TCP.

Each of these has advantages - the Protocol Buffers API is better for latency-sensitive workloads, while the Atom API gives more flexibility for cache design. However, both require a lot of work on each platform to implement.

To improve this situation, we have added a new default client protocol in EventStoreDB 20.6.0, using gRPC - a widely adopted standard which is interoperable across lots of platforms with minimal effort.

With EventStoreDB 20.6.0, we are shipping a .NET SDK for the new gRPC protocol.

Gossip over gRPC

With the preview releases of EventStoreDB 20.6.0, we've had great feedback from community members building out gRPC Clients and with that requested that we expose the cluster gossip protocol over gRPC for cluster aware clients to use.

The HTTP based gossip has not been removed and any existing operational tools and clients such the EventStoreDB .NET TCP Client will continue to work, however due to the role name change, the consumers of the protocol will have to be updated to use Follower instead of Slave and Leader instead of Master.

The EventStoreDB .NET TCP Client 5.x has been updated in 5.0.8 to understand the new cluster terminology.

Server-side filtering

We have also added the ability to filter both reads and subscriptions on the server-side. You can filter by either event type or stream name.

The intention here is to remove the need for having to use projections when you want to subscribe to EventStoreDB and only receive events from streams or events satisfying a particular filter.

  • New methods and overloads have been added for reads and subscriptions to the .NET TCP Client SDK.
  • New HTTP endpoints have been added as well for reading only.
  • The gRPC Client SDK only contains support for filtered subscriptions.

Read-Only Replicas

A common use case for adding more nodes to a cluster is being able to provide data from EventStoreDB closer to the destination where it's intended to be used. However, by adding more nodes to the cluster, you incur write latency as EventStoreDB requires a majority of the nodes in the cluster to acknowledge the write before it's deemed successful.

Previously EventStoreDB supported adding additional nodes to the cluster as Clones (which as of EventStoreDB 20.6.0 has been deprecated) for scaling out reads. This has led to a few issues as Clones can be promoted into the cluster which can mean that clients end up reading from a node that starts participating in normal quorum operations. This has two undesired side effects, in that additional load can be inadvertently placed on a quorum node, and in the event of network segregation the cluster can end up in a split brain situation.

To address both of these concerns, we have introduced the ability to mark a node as a Read-Only Replica which avoids these scenarios. This type of node will not participate in elections, and will not be promotable to a Clone, Follower or Leader node.

To start a node as a Read-Only Replica, the ReadOnlyReplica option has been introduced.

%> EventStore.ClusterNode.exe --read-only-replica

Both the EventStoreDB TCP and gRPC Clients have also been adjusted to allow the user to set the node preference to ReadOnlyReplica.

Leader Resignation

Performing maintenance work such as scavenging on a node can result in performance degradation. For example, scavenging is an I/O intensive operation, and running a scavenge on a leader node can affect the node's ability to serve write requests.

Providing the ability to resign the leader node gives customers the ability to perform maintenance work on a node without having to shutdown or remove the node from the cluster.

Resigning the leader node is currently a two-step process which leverages a previously available startup time feature called node priority. Two new admin HTTP endpoints have been added, priority to allow setting the node priority and resign to allow resigning the node.

To resign a leader node, the user has to issue two HTTP POST requests. The first action is to reduce the current leader node's priority so that during elections, another node with a higher priority will be chosen over the current leader node.

%> curl -X POST -d {} https://localhost:2113/admin/node/priority/-1 -u admin:changeit

The second action is to issue a resignation command which will explicitly start a round of elections.

%> curl -X POST -d {} https://localhost:2113/admin/node/resign -u admin:changeit

The above commands are privileged commands and require either a user in the $ops or $admins group to perform.

Restarting the projection subsystem

There are a few situations where the projection subsystem can get stuck and can often only be recovered by restarting the leader node. Restarting a node is not ideal as this interrupts read and write operations, and the node may take a while to start up again.

A new operation which allows you to just restart the projection subsystem has been added. This will stop and start the subsystem as if the node had been shut down, but without interrupting any other operations on the node. During the restart, the projections may enter a "suspended" state as they dispose of their subscriptions.

To restart the projection subsystem, a user has to issue an HTTP POST request to the new endpoint.

%> curl -X POST -d {} https://localhost:2113/projections/restart -u admin:changeit

The above command is a privileged command and requires either a user in the $ops or $admins group to perform.

Pinned by correlation id strategy for persistent subscriptions

A new competing consumer strategy has been added to persistent subscriptions, which presents events to the same consumer if they have the same correlation id.

In order to use this new strategy, the $by_correlation_id projection must be running, and you can now select this new strategy when creating the subscription through the UI or TCP client.

Liveness healthCheck

We have introduced a health check which can be queried via the {server_address}/health/live endpoint. This endpoint will return a 204 status code once EventStoreDB is ready to serve requests.

It is configured as the default health check in the Docker container.

Internal security in a cluster

The internal interfaces in a cluster are now secured by default and will no longer be externally accessible.

Internal TCP connections are mutually authenticated, and both follower and leader nodes will verify the certificates of the other nodes when a connection is made. This behaviour can be disabled using the --disable-internal-tcp-tls option, and is disabled by default in development mode (--dev).

In addition to the certificate requirement, the subject of the certificate (CN=eventstoredb-node) is also used to authorize a node as a system user.

New options for configuration of certificates

EventStoreDB now supports multiple options for configuring certificates to allow you to choose what best suits your setup. The options provided have also been updated to have more detailed descriptions and errors to help guide customers during setup.

A certificate can be provided from :

  • A PKCS #12 (.p12/.pfx) certificate file, by providing it to the --certificate-file option.
  • A public and private key pair, by setting --certificate-file to point to an X.509 (.pem, .crt, .cer, .der) certificate file, and pointing --certificate-private-key-file to a private key (.key).
  • The windows certificate store, using the --certificate-store-location, --certificate-store-name, --certificate-thumbprint and --certificate-subject-name options.

Max truncation safety feature

We have added a feature to avoid large unexpected truncations by adding a MaxTruncation setting that will stop any truncations over that size. This can be set via the following command-line argument, and can be disabled by setting it to -1.

--max-truncation=-1

The default is 1 chunk of data.

Ability to use host names in gossip and advertise endpoints

In the past, EventStoreDB has only supported using IP addresses in the gossip and advertise endpoints. We have now provided the ability to use hostname and port in the gossip seeds as well as the advertise endpoints.

Projection improvements

A number of improvements have been made to projections with a focus on stability and bug fixes.

This includes:

  • Exponential backoff with jitter when retrying writes.
  • A projection will retry more times before faulting.
  • Decreasing the number of writes projections perform.
  • Bug fixes, which can be found in the changelogs.

Cluster improvements

We have worked on improving cluster stability by reducing the number of elections triggered by normal operations:

  • Nodes that have just started up will first try and identify and subscribe to an existing leader instead of triggering elections. This can be helpful during maintenance and upgrades.
  • When a node loses TCP connection to the leader, it will issue a gossip and determine whether the leader is dead or not based on if it gets a response.
  • Information about the epoch has been added to election messages to allow nodes to make better decisions during elections.

This should result in fewer leader change overs and fewer offline truncations. We have also updated the cluster replication to ensure that dirty reads from followers are no longer possible.