More often than not, the creation of extensions (LSXs/uDSMs) for unsupported devices/log sources for QRadar is a straightforward stepwise procedure as described @ https://qmmunity.q1labs.com/node/1130 However, in some cases the implementation becomes rather challenging.
Based on our QRadar LSX/uDSM development experience, we have singled out three basic implementation scenarios:
1) Supporting Syslog or plain-text log sources
Typical implementation scenario for Syslog or plain-text based devices requires one to two days of development at a very minimum and includes the following steps:
- audit data structure investigation;
- creation of parsing rules;
- mapping custom events to QRadar categories;
- testing;
- creation of custom reports and correlation rules.
This is the simplest but not the most common scenario. The amount of development efforts often depends on the number of unique event types for a particular device. Your work consists mostly of applying regular expressions and assigning QIDs (mapping).
2) Supporting log sources by means of existing collect protocols
Another implementation scenario makes use of an existing collect protocol (i.e. JDBC) that requires additional pre-configuration steps. In case of a database, in addition on typical implementation steps mentioned above you will have to investigate the database structure and create views to combine all required data. Sometimes it might be also necessary to create shell/batch wrapper scripts.
3) Supporting a log source with an unsupported protocol
The most complicated scenario is related to supporting a new device with an unsupported protocol. This includes application-specific binary log files, access via API, multiline logs, or database without a JDBC access option (e.g. Paradox or Lotus). This scenario dramatically increases the amount of required efforts due to the following activities in addition to Scenario 1:
- investigation of the target platform; you need to find out how to configure the audit, where the audit information is stored, what is the format of the audit data and how you can extract it;
- investigation of third-party or application-specific tools to access audit information (the worst case is when you have to create your own extraction tool using application-specific languages, like in the case of 1C, a business application suite widely popular in Russian-speaking countries;
- creation of shell/batch wrapper scripts to extract data on a recurrent basis without duplicates and data loss;
- configuration of an applicable QRadar protocol to feed data to QRadar.
In IT, the term “event” is often used casually, synonymous with any log entry (i.e. anything that occurs at a particular time, and has a time stamp associated with it). But based on this definition, an event may be something as simple as accepting a single arbitrary packet into the network. It may even be a “heartbeat,” issued by some program at some arbitrary interval ranging from once each day to once each microsecond. Consider the extreme case, where an event is defined as the high-speed system clock of a computer ticking one count. It is certainly classifiable as an “event,” but is anyone really interested in logging this for each device, one million or more times each second from every managed device?
Given that, when the PCI-DSS standard refers to the requirement of logging all events on the network, it cannot possibly be referring to the logging of each and every conceivable event. We must interpret that the standard be referring only to “significant events”. As any operator of a SIEM system will tell you that most of what you are logging is pretty much junk anyway. And, you will be required to keep this data for several years. The data will be as voluminous as possible (to satisfy auditors, but also forensics) and each passing year will make that operator more appreciative of how low-cost disk storage has become.
Within the terabytes worth of log data you have collected, there may be a few “significant events.” Hopefully, that number will be low, optimally zero. The other data is okay to log (actually required by auditors) but this data is not necessarily “event data.” It is simply data to support and clarify the more particular and important “event data,” which is what you need for SIEM to be at its most effective state.
Event data is a highly specific subclass of log data. It is has various characteristics that distinguish it from other log data.
Specifically:
- Pertinence. An event is distinguished from other log data based upon its relevance. How interesting is this event? How often does it occur? (It is important to note that, according to the basic tenets of information theory, the relevance of an event goes down the more often it occurs. An event that occurs regularly is not very relevant.)
- Context. The event must have enough of a context to make it understandable. For example, the statement “something very important happened at noon today” has no specific context. It could not be classified as an event by our definition. It may be pertinent (possibly based upon the source of this data), but there is not enough information to call this an event. Contrast the statement with something more specific, such as “You won the state lottery today at noon.” The latter statement is definitely an event, because you now know the context.
- Timeliness. Information cannot really be classified as an event unless it can be associated with a fairly precise time. The more timely the message, the more likely it can be considered an event. For example, stating that some long expected movie release will occur in the fall does not make that film into an event. It is simply information about an event. The actual event occurs at the precise day of the release, which in this case is the eventful movie premier. It has a specific time associated with it. In the best case, the event occurs in real-time.
- Actionability. If information is pertinent, and timely, it must still be actionable to be useful. The event requires something else that logically follows – perhaps further investigation, or corrective action. If a log message is not actionable, it can't really be classified as an event. Note that the event may not actually precipitate an action, but should still be actionable. For example, your birthday is an actionable event, even if you choose not to throw a party.
