Welcome!

Linux Containers Authors: Pat Romanski, Elizabeth White, Dana Gardner, Carmen Gonzalez, Yeshim Deniz

Related Topics: @BigDataExpo, Java IoT, Microservices Expo, Linux Containers, Agile Computing, @CloudExpo

@BigDataExpo: Article

What Is 'Real-Time' Anyway...?

One of my (least) favorite buzzwords in the logging space is “real-time."

I love a good buzzword...cloud, Big Data, analytics... And even more than the buzzwords, I love the liberties people tend to take applying these buzzwords to their new systems and services. Such buzzwords regularly get abused and often get washed into marketing material and product websites in an attempt to hoodwink and woo new unsuspecting customers. One of my (least) favorite buzzwords, that I've noticed popping up more recently in particular in the logging space is "real-time."

So what does real-time mean anyway? Like with all good computer science questions, it really depends on the context.  John Stankovic, in his seminal 1988 article in IEEE Computer entitled ‘Misconceptions of real-time computing' describes a real time system as follows:

"A real-time system is one in which the correctness of the system depends not only on the logical result of computation, but also on the time at which the results are generated."[1]

An example is, what are referred to as, "hard-real-time systems" [2], where computation must meet stringent timing constraints and one must guarantee that those computations must be completed before specified deadlines. Failure to do so can lead to intolerable system degradation and can in some applications lead to catastrophic loss of life or property.

Many safety-critical systems are hard-real-time systems, and include embedded tactical systems for military applications, flight mission control, traffic control, production control, nuclear plant control etc. and in many cases the real-time properties need to be guaranteed and proven - often using techniques such as formal methods [3] for example.

Near real-time is often defined as not having any hard constraints, but implies that there are no significant delay in processing of the event. In many cases this means within a few milliseconds or seconds of the event - again this really depends on the context. From a system monitoring perspective (of non-safety critical applications) near real-time, i.e. within a few seconds, is usually sufficient when it comes to alerting for example. And by non-safety critical, I mean, your site/app might be down, but it will not lead to loss of life. That being said it could be resulting in serious loss of $$$.

In such scenarios if there is something awry with one of system components you want to be notified about this immediately, so that right away you can go about rectifying the issue. A few minutes is usually unacceptable however, as generally this means that users/customers of your system/service are being effected without you knowing anything about this, resulting in damage to your brand and business top line.

This is what really surprises me when I see so many log management solutions that run their alerting as background jobs or saved searches which run periodically every 5, 10 or 15 mins. In my opinion this doesn't really cut it when it comes to alerting and is NOT real- time/near real-time by any standard. Consider this, if there was an emergency at home do you think it would be acceptable to wait 5, 10 or 15 minutes before you picked up the phone and called the emergency services. A few seconds, yes, a few minutes NO!

A further observation by Stankovic in his 1988 article was that another common misconception in relation to real-time systems is that throwing hardware at the problem can solve this issue - however as Stankovic rightly points out, throwing hardware at the problem is not the answer - it's all about the architecture.

That's why we have built a very different architecture to any other logging provider. In short, most log management solutions work as follows:

  • Data is sent to the log management service
  • It is indexed and written to disk
  • You can make use of their (complex) search language to dig into your data
  • If you want to create notifications you need to set up saved searches (using the complex search language) that you can schedule to run every 5, 10 or 15 mins.

At Logentries we have flipped this approach on its head and have built a unique pre-processing layer as part of our system architecture that allows for real-time processing of your data such that the analysis of your log events is done up front and in near real-time. And, when we say real-time, we mean real-time.

It works as follows:

LIVE_TAIL

  • Data is sent to Logentries
  • It passes through our pre-processing layer which analyzes each event for defined patterns(i.e.keywords,regular expressions,defined search expressions) in real-time
  • Notifications can be generated from our pre-processing layer such that you receive them within seconds of the important events occurring (e.g. exceptions, errors, warnings...)

The end result is that you get notified in seconds as opposed to minutes. In a world where time is money, and where buzzwords are only as useful as the architecture behind them, I vote for REAL real-time alerting - as it is an important requirement in any logging service!

[1] J. Stankovic, Misconceptions of real-time computing', IEEE Computer, 1988

[2] Jia Xu and David Lorge Parnas, ‘On Satisfying Timing Constraints in Hard-Real-Time Systems' IEEE Transactions on Software Engineering, Vol 19, No. 1, Jan 1993

[3] ‘Formal methods for the design of Real Time Systems', Springer, International School on Formal Methods for the Design of Computer, Communication and Software Systems, September 2004

More Stories By Trevor Parsons

Trevor Parsons is Chief Scientist and Co-founder of Logentries. Trevor has over 10 years experience in enterprise software and, in particular, has specialized in developing enterprise monitoring and performance tools for distributed systems. He is also a research fellow at the Performance Engineering Lab Research Group and was formerly a Scientist at the IBM Center for Advanced Studies. Trevor holds a PhD from University College Dublin, Ireland.

@ThingsExpo Stories
Unsecured IoT devices were used to launch crippling DDOS attacks in October 2016, targeting services such as Twitter, Spotify, and GitHub. Subsequent testimony to Congress about potential attacks on office buildings, schools, and hospitals raised the possibility for the IoT to harm and even kill people. What should be done? Does the government need to intervene? This panel at @ThingExpo New York brings together leading IoT and security experts to discuss this very serious topic.
It is one thing to build single industrial IoT applications, but what will it take to build the Smart Cities and truly society changing applications of the future? The technology won’t be the problem, it will be the number of parties that need to work together and be aligned in their motivation to succeed. In his Day 2 Keynote at @ThingsExpo, Henrik Kenani Dahlgren, Portfolio Marketing Manager at Ericsson, discussed how to plan to cooperate, partner, and form lasting all-star teams to change the...
The buzz continues for cloud, data analytics and the Internet of Things (IoT) and their collective impact across all industries. But a new conversation is emerging - how do companies use industry disruption and technology enablers to lead in markets undergoing change, uncertainty and ambiguity? Organizations of all sizes need to evolve and transform, often under massive pressure, as industry lines blur and merge and traditional business models are assaulted and turned upside down. In this new da...
SYS-CON Events announced today that Hitrons Solutions will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Hitrons Solutions Inc. is distributor in the North American market for unique products and services of small and medium-size businesses, including cloud services and solutions, SEO marketing platforms, and mobile applications.
“We're a global managed hosting provider. Our core customer set is a U.S.-based customer that is looking to go global,” explained Adam Rogers, Managing Director at ANEXIA, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
WebRTC services have already permeated corporate communications in the form of videoconferencing solutions. However, WebRTC has the potential of going beyond and catalyzing a new class of services providing more than calls with capabilities such as mass-scale real-time media broadcasting, enriched and augmented video, person-to-machine and machine-to-machine communications. In his session at @ThingsExpo, Luis Lopez, CEO of Kurento, introduced the technologies required for implementing these idea...
@GonzalezCarmen has been ranked the Number One Influencer and @ThingsExpo has been named the Number One Brand in the “M2M 2016: Top 100 Influencers and Brands” by Onalytica. Onalytica analyzed tweets over the last 6 months mentioning the keywords M2M OR “Machine to Machine.” They then identified the top 100 most influential brands and individuals leading the discussion on Twitter.
In an era of historic innovation fueled by unprecedented access to data and technology, the low cost and risk of entering new markets has leveled the playing field for business. Today, any ambitious innovator can easily introduce a new application or product that can reinvent business models and transform the client experience. In their Day 2 Keynote at 19th Cloud Expo, Mercer Rowe, IBM Vice President of Strategic Alliances, and Raejeanne Skillern, Intel Vice President of Data Center Group and G...
Financial Technology has become a topic of intense interest throughout the cloud developer and enterprise IT communities. Accordingly, attendees at the upcoming 20th Cloud Expo at the Javits Center in New York, June 6-8, 2017, will find fresh new content in a new track called FinTech.
20th Cloud Expo, taking place June 6-8, 2017, at the Javits Center in New York City, NY, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy.
Manufacturers are embracing the Industrial Internet the same way consumers are leveraging Fitbits – to improve overall health and wellness. Both can provide consistent measurement, visibility, and suggest performance improvements customized to help reach goals. Fitbit users can view real-time data and make adjustments to increase their activity. In his session at @ThingsExpo, Mark Bernardo Professional Services Leader, Americas, at GE Digital, discussed how leveraging the Industrial Internet and...
SYS-CON Events announced today that delaPlex will exhibit at SYS-CON's @CloudExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. delaPlex pioneered Software Development as a Service (SDaaS), which provides scalable resources to build, test, and deploy software. It’s a fast and more reliable way to develop a new product or expand your in-house team.
Web Real-Time Communication APIs have quickly revolutionized what browsers are capable of. In addition to video and audio streams, we can now bi-directionally send arbitrary data over WebRTC's PeerConnection Data Channels. With the advent of Progressive Web Apps and new hardware APIs such as WebBluetooh and WebUSB, we can finally enable users to stitch together the Internet of Things directly from their browsers while communicating privately and securely in a decentralized way.
"Matrix is an ambitious open standard and implementation that's set up to break down the fragmentation problems that exist in IP messaging and VoIP communication," explained John Woolf, Technical Evangelist at Matrix, in this SYS-CON.tv interview at @ThingsExpo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
Web Real-Time Communication APIs have quickly revolutionized what browsers are capable of. In addition to video and audio streams, we can now bi-directionally send arbitrary data over WebRTC's PeerConnection Data Channels. With the advent of Progressive Web Apps and new hardware APIs such as WebBluetooh and WebUSB, we can finally enable users to stitch together the Internet of Things directly from their browsers while communicating privately and securely in a decentralized way.
The explosion of new web/cloud/IoT-based applications and the data they generate are transforming our world right before our eyes. In this rush to adopt these new technologies, organizations are often ignoring fundamental questions concerning who owns the data and failing to ask for permission to conduct invasive surveillance of their customers. Organizations that are not transparent about how their systems gather data telemetry without offering shared data ownership risk product rejection, regu...
The security needs of IoT environments require a strong, proven approach to maintain security, trust and privacy in their ecosystem. Assurance and protection of device identity, secure data encryption and authentication are the key security challenges organizations are trying to address when integrating IoT devices. This holds true for IoT applications in a wide range of industries, for example, healthcare, consumer devices, and manufacturing. In his session at @ThingsExpo, Lancen LaChance, vic...
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo 2016 in New York. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place June 6-8, 2017, at the Javits Center in New York City, New York, is co-located with 20th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry p...
SYS-CON Media announced today that @WebRTCSummit Blog, the largest WebRTC resource in the world, has been launched. @WebRTCSummit Blog offers top articles, news stories, and blog posts from the world's well-known experts and guarantees better exposure for its authors than any other publication. @WebRTCSummit Blog can be bookmarked ▸ Here @WebRTCSummit conference site can be bookmarked ▸ Here
In his keynote at @ThingsExpo, Chris Matthieu, Director of IoT Engineering at Citrix and co-founder and CTO of Octoblu, focused on building an IoT platform and company. He provided a behind-the-scenes look at Octoblu’s platform, business, and pivots along the way (including the Citrix acquisition of Octoblu).