BOSTON, Oct. 31, 2013 (GLOBE NEWSWIRE) -- Researchers at Logentries, a leading SaaS provider for collecting and analyzing huge quantities of machine-generated log data, found that less than one percent of log data generated by applications built on the Heroku Platform-as-a-Service (PaaS) contain important application errors and exceptions that help DevOps professionals improve health and performance of their applications. The research, Big Insights from Little Data: A Spotlight on Unlocking Insights from the Log Data That Matters, is an analysis of 22 billion log events across more than 6,000 Heroku applications. The study highlights the challenge of looking for specific critical information buried within large amounts of big data. In recognition of this reality, Logentries has released its first third-party product add-ons to further extend its capabilities for unlocking insights from log data.
"Our research highlights the modern equivalent of the needle in the cloud-enabled, haystack. While often overlooked or missed entirely, error codes are an essential part of understanding and improving the overall health and performance of Heroku applications," said Dr. Trevor Parsons, Co-Founder and Chief Scientist at Logentries. "When critical information is buried in billions of log events identifying such problems can be a major challenge."
It's the Little Data That Counts
In analyzing 22 billion Heroku log events from more than 6,000 Heroku applications from the perspective of a DevOps professional looking to troubleshoot, diagnose and monitor Heroku applications, Logentries found that:
· 99.82% of log events were signal noise for the defined DevOps use case, with 0.18% of log events containing critical information for DevOps (i.e. application/platform errors, warnings or exceptions defined in collaboration with Heroku engineers).
· Within the 0.18% of events containing critical information, 0.08% were warnings, 0.05% were critical events, 0.01% were fatal and 0.04% were application exceptions.
The report also shows this issue is significantly exasperated as systems grow in size, to where only 0.02% of log data contained valuable information for the DevOps use case (99.98% noise) in samples of log data from larger applications. The report also provides a more fine-grained analysis of the breakdown of this data and gives insight into the characteristics of the log data produced by Heroku applications of varying sizes.
Given that 0.18% of log data contains valuable information, the survey demonstrates that while big data receives most of the attention, it's actually the little data that provides the important insights. The research findings highlight the challenge for those responsible digging through the machine-generated log data to find this information.
"Heroku provides developers and partners with a very powerful platform as a service to build applications without having to worry about the underlying infrastructure and services," said Robbie Thng, Add-ons Product Manager at Heroku. "Logentries not only highlights the importance of having visibility and insights into all aspects of your log data but also provides unique capabilities to assist Heroku customers to improve their applications and Heroku partners to enhance their services."
Adept Mobile, a Boston-based company that builds and manages web applications for major U.S. sports franchises, understands the problem of dealing with large amounts of log data: "At Adept, every day can be game day, when we need to ensure our Heroku applications are exceeding expectations with every user session," said Craig Heneveld, Director of Technology at Adept Mobile. "Whether it's UFC 170 or an NFL playoff game, we rely on Logentries to provide us with quick and easy access to the individual errors, application exceptions and other log events, without wading through unnecessary information that isn't relevant to our needs."
Logentries New Add-Ons Speed "Time to Insight"
In conjunction with the study release, Logentries unveiled its first 3rd party product add-ons designed to further combat the challenge posed by the abundance of signal noise in log data. These add-ons, which are preconfigured to work with Logentries, provide users with instant insights for the Heroku platform, including:
· Powerful Database-as-a-Service with Heroku Postgres - Users can view Heroku Postgres log errors and performance issues in a single dashboard and perform deep dives to investigate specific errors automatically tagged by Logentries.
· Automated Scaling of Heroku Dynos with AdeptScale - Users can automatically log Adept Scale events to Logentries and easily evaluate all of their log entries in a single location.
· Run RabbitMQ as a Service with CloudAMQP - Users can decouple, distribute and scale their Heroku applications with a single, common source of log data, and view preconfigured event tags for RabbitMQ, all within Logentries.
"As the roles and philosophies between developers and IT operations continue to come closer together, the importance of log data to improve visibility and collaboration across the application lifecycle is increasing," said Andrew Burton, CEO of Logentries. "Our goal is not only to make log data simply accessible for anyone, but to go beyond the inherent limitations of searching for information and to unlock key insights through preprocessing and pre-tagging of critical information as it's captured."
The full research can be downloaded at Logentries.com, http://www.logentries.com/insights/research.
More information on the Logentries Add-Ons can also be found by visiting Logentries at http://logentries.com/insights/addons.
About Logentries
Logentries (http://logentries.com) is a SaaS solution for collecting and analyzing huge quantities of machine-generated log data and making that data easily accessible to individual developers, small teams, and enterprise customers. While traditional log management solutions require advanced technical skills to use or are costly to setup, Logentries provides a simply accessible alternative. With Logentries your log files are filtered, pre-processed and correlated up front for quicker and easier retrieval of the log entries that matter most. In turn, this pre-processing is combined with a collective insights model that enables important information to be dynamically tagged, automatically routed, and easily shared across teams and computing platforms. With over 1,000 customers in a 100 countries, Logentries is processing more than 10 billion events a day.