03Mar, 2021
How to reduce one or many Sitecore log entries from Application Insights
One thing you'll be familiar with in Sitecore Managed Cloud is Application Insights, which is going to give you access to the log lines Sitecore produces. My first impression of this was, how do I sift through all this garbage? There's a lot of data getting sent to AI, and we only need about 5% of it!
You can use the filtering options provided to you in Application Insights. For instance, in the screenshot below I've filtered down to the instance with a CM Role.
However, this doesn't reduce the number of entries sent to AI, and there's a limit to what you can store (or increase your budget and we don't want that). When this limit is exceeded, you'll see the message "Data received from your application is being sampled to reduce the volume of telemetry data retained; only sampled documents will be returned. The sampling may be applied by the Application Insights SDK or on ingestion by Application Insights.”, and that's not cool. I'm dependent on seeing certain logs should I want to troubleshoot, so let's make sure this sampling issue never comes up.
The log4net StringMatchFilter can do the trick, as you can have some entries skip writing to AI. Notice how this example has “Job started”, since that, and “Job ended” take up pretty much most of our logs, and I doooon't care!
<appender name="LogFileAppender" type="log4net.Appender.SitecoreLogFileAppender, Sitecore.Logging">
<filter type="log4net.Filter.StringMatchFilter">
<stringToMatch value=" Job started" />
<acceptOnMatch value="false" />
</filter>
</appender>
This will work for a single string, but if you try to repeat the log4net.Filter.StringMatchFilter (I'm looking at you, Job ended) you'll just have the final one removed. Regex to the rescue! This example will allow for all the string matches you could want. Let's get those Job entries outta here, and throw in a couple more for good measure. What a world.
<appender name="LogFileAppender" type="log4net.Appender.SitecoreLogFileAppender, Sitecore.Logging">
<filter type="log4net.Filter.StringMatchFilter">
<regexToMatch value="^.*(Job started|Job ended|document not found page|Cache created).*$" />
<acceptOnMatch value="false" />
</filter>
</appender>
All you need to do it patch this into your instances and refresh AI so see that log volume drop drastically.
