Margaret Rouse is an award-winning technical writer and teacher known for her ability to explain complex technical subjects simply to a non-technical, business audience. Over…
Apache Kafka is an open-source publish-subscribe message system designed to provide quick, scalable and fault-tolerant handling of real-time data feeds. Unlike traditional enterprise messaging software, Kafka is able to handle all the data flowing through a company, and to do it in near real time.
Kafka is written in Scala and was originally developed by LinkedIn. Since that time, a number of companies have used it to build real-time platforms.
Kafka has many similarities to transaction logs, and it maintains feeds of messages in topics. Producers write data to topics and consumers read from those topics, which are partitioned and replicated across multiple nodes in a distributed system format. Kafka is unique in that it treats each topic partition as a log, and each message in a partition is assigned a unique offset. It retains all messages for a certain amount of time, and consumers are responsible for tracking their location in each log. This differs from previous systems, where brokers were responsible for this tracking, which severely limited the system’s ability to scale as the number of consumers increased. This structure allows Kafka to support many consumers and retain large amounts of data with very low overhead.
Kafka can be used:
Kafka can be used alongside Apache Storm, Apache HBase and Apache Spark for real-time analysis and rendering of streaming data.
Techopedia’s editorial policy is centered on delivering thoroughly researched, accurate, and unbiased content. We uphold strict sourcing standards, and each page undergoes diligent review by our team of top technology experts and seasoned editors. This process ensures the integrity, relevance, and value of our content for our readers.
Margaret is an award-winning technical writer and teacher known for her ability to explain complex technical subjects to a non-technical business audience. Over the past twenty years, her IT definitions have been published by Que in an encyclopedia of technology terms and cited in articles by the New York Times, Time Magazine, USA Today, ZDNet, PC Magazine, and Discovery Magazine. She joined Techopedia in 2011. Margaret's idea of a fun day is helping IT and business professionals learn to speak each other’s highly specialized languages.
What is Differential Privacy? Differential privacy is a mathematical framework for determining a quantifiable and adjustable level of privacy protection....
Margaret RouseTechnology Expert
What is cPanel Used For? cPanel is a crucial tool to help you access hosting features via a simple, non-technical...
Ilijia MiljkovacTechnology Writer
What is Operational Technology? Operational Technology, or OT, refers to the hardware and software systems that are used to control...
Marshall GunnellIT & Cybersecurity Expert
Trending NewsLatest GuidesReviewsTerm of the Day