How do you create a secure, resilient, effective, compliant and cost efficient syslog event management infrastructure?
If you are asking yourself this question, its likely that by this stage you have invested in a market leading SIEM platform such as Splunk, QRadar, LogRhythm, Elastic or any other of the market leaders – maybe you have adopted a multi-tier approach and have more than one of the listed.
It’s not uncommon to have hundreds of thousands of events per second flowing across enterprise networks and given that the majority are sent via UDP and hence connectionless, it’s possible that in times of peak network capacity that some events are simply lost. Other considerations to bear in mind are things such as security, compliance and volume base license costs. We will highlight some use cases in this post to show ways in which you can address these considerations, creating an efficient, secure, GDPR compliant and cost effective syslog environment.
Challenge#1: Platform Agnostic Log Management
Challenges:
Variety of sources & schema, multiple destinations, delivery guarantee, fault tolerance
End Goals:
Unified collection, real-time transformation, secure transit, buffering/caching
Here, Syslog-ng relays and PE have been deployed to deliver syslog messages using RLTS protocol which guarantees log delivery at different locations, should a log be dropped it is resent until it is acknowledged and forwarded. PE also uses TLS to secure the delivery, meaning that sensitive information cannot be accessed by third parties. Syslog-ng also caches and buffers on the local disk should the network or device connection become unavailable. Now multiple teams can retrieve logs for their specific use cases in a standardised, efficient and secure manner.
Challenge #2: Long-Term Storage & Search
Challenges:
Usage based licensing costs, storage costs, varying retention requirements
End Goals:
Implement long-term storage layer, automated retention policies, automated archiving, indexed and compressed
In this case, relays have been used in conjunction with the Syslog Store Box (SSB). Policies have been created to index the events at the SSB and send to SAN for lower cost storage. Also, content such as human readable content has been stripped and the logs re-written to further drive efficiency and reduce SIEM license costs along with de-duplication. Logs are indexed and compresses resulting in faster searching in archives. Policies created to create differing retention policies as required.
Challenge #3: Advanced Routing / Filtering
Challenges:
Varying events-per-second EPS, usage based licensing, usage based planning, cost implications
End Goals:
Filter out irrelevant data, asset based planning, cost effective retention
Here, relays and PE have been used in conjunction with advanced filtering techniques and directional forwarding that have drastically reduced the volume of messages hitting the SIEM and compliance platforms. Syslog-ng can be deployed as an agent on a wide number of hosts and flexibly route content to multiple destinations without the need to deploy multiple agents on devices. Duplicates and unnecessary content has been stripped and discarded.
Challenge #4: Compliance, Even Without a SIEM
Challenges:
unsorted, mixed content, too noisy for correlation, mixed compliance models, unfiltered data exposed
End Goals:
Logically separate content, data can be filtered further, different archiving policies, repository access controls, user specific log ‘views’
Using the SSB, log data is stored in encypted, compressed and time stamped binary files with access restricted to authorized personnel only. Authentication, authorisation and accounting settings provide granular control based on user privileges. Can also be integrated with LDAP and Radius database. Ability to store up to 10TB of uncompressed data in the largest SSB. PE also ensures that messages cannot be accessed by third parties by using TLS to encrypt the communication between agents and syslog-ng store box. Filtering and routing rules send data to correct platforms.
Challenge #5: Reliable Log Infrastructure
Challenges:
Critical event, logs never sent, connection saturation, single point of failure, data open to inspection
Solution:
Relays as a local staging post, consolidate cache & buffer, alternate destinations, RLTP: received and understood, encryption & timestamping
A combination of Syslog-ng Agents, relays and SSB have been used to create TLS encrypted messages sent through the network securely. RLTP has been used to guarantee log delivery using received and understood methodology. Logs stored in encrypted state on the SSB, indexed and timestamped and sent to SAN and SIEM for processing.
Thanks & credit to the team @ BALABIT & OneIdentity for assistance with content.
If you would like to discuss how these solutions can transform your syslog architecture, help you to meet GDPR compliance regulations and save you substantially on your license costs please get in touch.
