Getting began with Kafka consumer metrics

Apache Kafka stands as a well known open supply tournament retailer and movement processing platform. It has advanced into the de facto usual for information streaming, as over 80% of Fortune 500 firms use it. All primary cloud suppliers supply controlled information streaming services and products to satisfy this rising call for.

One key good thing about choosing controlled Kafka services and products is the delegation of duty for dealer and operational metrics, permitting customers to focal point only on metrics explicit to packages. In this text, Product Manager Uche Nwankwo supplies steerage on a suite of manufacturer and person metrics that consumers must observe for optimum efficiency.

With Kafka, tracking usually comes to more than a few metrics which are associated with subjects, walls, agents and person teams. Standard Kafka metrics come with knowledge on throughput, latency, replication and disk utilization. Refer to the Kafka documentation and related tracking equipment to know the particular metrics to be had on your model of Kafka and methods to interpret them successfully.

Why is it vital to observe Kafka purchasers?

Monitoring your IBM® Event Streams for IBM Cloud® example is a very powerful to verify optimum capability and general well being of your information pipeline. Monitoring your Kafka purchasers is helping to spot early indicators of utility failure, similar to excessive useful resource utilization and lagging shoppers and bottlenecks. Identifying those caution indicators early allows proactive reaction to doable problems that reduce downtime and save you any disruption to industry operations.

Kafka purchasers (manufacturers and shoppers) have their very own set of metrics to observe their efficiency and well being. In addition, the Event Streams carrier helps a wealthy set of metrics produced via the server. For additional information, see Monitoring Event Streams metrics by using IBM Cloud Monitoring.

Client metrics to observe

Producer metrics

Metric	Description
Record-error-rate	This metric measures the typical per-second selection of data despatched that ended in mistakes. A excessive (or an building up in) record-error-rate would possibly point out a loss in information or information no longer being processed as anticipated. All those results would possibly compromise the integrity of the information you might be processing and storing in Kafka. Monitoring this metric is helping to make sure that information being despatched via manufacturers is appropriately and reliably recorded to your Kafka subjects.
Request-latency-avg	This is the typical latency for every produce request in ms. An building up in latency affects efficiency and would possibly sign a subject. Measuring the request-latency-avg metric can lend a hand to spot bottlenecks inside your example. For many packages, low latency is a very powerful to verify a top quality consumer enjoy and a spike in request-latency-avg would possibly point out that you’re attaining the bounds of your provisioned example. You can repair the problem via converting your manufacturer settings, as an example, via batching or scaling your plan to optimize efficiency.
Byte-rate	The moderate selection of bytes despatched in keeping with moment for a subject matter is a measure of your throughput. If you movement information incessantly, a drop in throughput can point out an anomaly to your Kafka example. The Event Streams Enterprise plan begins from 150MB-per-second break up one-to-one between ingress and egress, and it’s important to understand how a lot of that you’re eating for efficient capability making plans. Do no longer move above two-thirds of the utmost throughput, to account for the imaginable affect of operational movements, similar to inner updates or failure modes (as an example, the lack of an availability zone).

Scroll to view complete desk

Table 1. Producer metrics

Consumer metrics

Metric	Description
Fetch-rate fetch-size-avg	The selection of fetch requests in keeping with moment (fetch-rate) and the typical selection of bytes fetched in keeping with request (fetch-size-avg) are key signs for a way neatly your Kafka shoppers are appearing. A excessive fetch-rate would possibly sign inefficiency, particularly over a small selection of messages, because it way inadequate (perhaps no) information is being gained every time. The fetch-rate and fetch-size-avg are suffering from 3 settings: fetch.min.bytes, fetch.max.bytes and fetch.max.wait.ms. Tune those settings to reach the required general latency, whilst minimizing the selection of fetch requests and probably the weight at the dealer CPU. Monitoring and optimizing each metrics guarantees that you’re processing information successfully for present and long run workloads.
Commit-latency-avg	This metric measures the typical time between a dedicated list being despatched and the devote reaction being gained. Similar to the request-latency-avg as a manufacturer metric, a strong commit-latency-avg implies that your offset commits occur in a well timed means. A high-commit latency would possibly point out issues throughout the person that save you it from committing offsets temporarily, which immediately affects the reliability of information processing. It would possibly result in replica processing of messages if a shopper should restart and reprocess messages from a up to now uncommitted offset. A high-commit latency additionally way spending extra time in administrative operations than precise message processing. This factor would possibly result in backlogs of messages ready to be processed, particularly in high-volume environments.
Bytes-consumed-rate	This is a consumer-fetch metric that measures the typical selection of bytes ate up in keeping with moment. Similar to the byte-rate as a manufacturer metric, this must be a strong and anticipated metric. A surprising exchange within the anticipated development of the bytes-consumed-rate would possibly constitute a subject along with your packages. A low fee may well be a sign of potency in information fetches or over-provisioned assets. A better fee would possibly crush the patrons’ processing capacity and thus require scaling, growing extra shoppers to steadiness out the weight or converting person configurations, similar to fetch sizes.
Rebalance-rate-per-hour	The selection of crew rebalances participated in keeping with hour. Rebalancing happens each time there’s a new person or when a shopper leaves the crowd and reasons a prolong in processing. This occurs as a result of walls are reassigned making Kafka shoppers much less environment friendly if there are a large number of rebalances in keeping with hour. A better rebalance fee in keeping with hour will also be led to via misconfigurations resulting in volatile person conduct. This rebalancing act could cause an building up in latency and would possibly lead to packages crashing. Ensure that your person teams are strong via monitoring a low and strong rebalance-rate-per-hour.

Scroll to view complete desk

Table 2. Consumer metrics

The metrics must duvet all kinds of packages and use instances. Event Streams on IBM Cloud supply a wealthy set of metrics which are documented right here and can supply additional helpful insights relying at the area of your utility. Take the next move. Learn extra about Event Streams for IBM Cloud.

What’s subsequent?

You’ve now were given the information on very important Kafka purchasers to observe. You’re invited to place those issues into follow and take a look at out the absolutely controlled Kafka providing on IBM Cloud. For any demanding situations in arrange, see the Getting Started Guide and FAQs.

Learn extra about Kafka and its use instances

Provision an instance of Event Streams on IBM Cloud

Was this text useful?

YesNo

Product Manager, Event Streams on IBM Cloud