By far the biggest and most common mistake most companies make when it comes to security monitoring is they monitor too much. Just because you can does not mean you should.
Modern IT systems create huge amounts of data and metrics to monitor, and most of the modern Security Incident & Event Management (SIEM) platforms are designed to hold large amounts of data, with fast built in searching. So whats the problem.
To start with their are considerable financial costs to transfer and store this data. For compliance reasons many companies are required to keep log data for extended periods of time usually around 5 years. Even smaller companies can easily generate 50GB of log data daily. This adds up quickly.
Although the financial cost is important its not the most important factor. The reason we use SIEM platforms is because its nearly impossible for even large security teams to identify security events by looking at large log files and impossible to do in real time. There is simply too much data. When we try to monitor and alert everything we create the same problem just too much noise, and real incidents often get lost or ignored because of the noise.
So how to we solve this problem?
We need to look at a few factors, the business, infrastructure and real risks. Once we have defined these areas we can identify the priority items and only alert on these threats. When it comes to real time alerts, less is really more. A good rule of thumb is if your SIEM platform generates an alert but the security team does not take any action then you should kill the alert as its just noise. If the alert is marked for review at a future date then those alerts should be grouped together in a weekly or monthly report. You only wont to receive real time alerts that are investigated by the security team.
security monitoring needs to be continually fine tuned and adapted as things change to stay relevant. I like to think of security monitoring as more of an art than a science and takes years of experience and a good understanding of the business risks to be setup correctly.
You simply cant afford to miss a real attack or security breach, keep it simple and always make sure you are looking at your top priority threats.