I've run into a bit of a snag and I'm not sure how to get around it. Here's the situation...
This is a device that accumulates syslog messages, dumps them to files and forwards them on to other security monitoring devices. The file it creates is eaten by Splunk and indexed so users can run queries against the logs, create reports, etc. Normally this works great. In this case, the customer has a couple devices, AIX servers, that are sending logs with their local time in the timestamp (GMT +1). Not only that, but the logs don't reference the local timezone. Syslog-ng assumes the timestamp is the local time (GMT), so it applies its own header with the exact same timestamp, but as GMT. When Splunk "eats" the logs, it shows these messages as being 1hr into the future. If you search for something that arrives right now, you won't find it, but if you wait an hour, it appears. Or, if you search for "All Time", it appears as the first result (Splunk displays most recent events first). Yup, we've created a time machine. :)
The easy fix is to add a configuration to Splunk telling it what TZ these hosts are in, but that still leaves the raw logs on disk with the bad timestamp. Normally this isn't a problem, but if you needed to work with the raw logs, you would have to account for the 1hr offset when correlating events across these two hosts.
You can apply TZ adjustments via syslog-ng, but I only need to apply it to these two hosts. According to the docs, if you add the time_zone() modifier to syslog-ng, it will only use it when no TZ is specified in the message. I can't do that, but I'm not absolutely certain the TZ is listed in all the other host logs. They are working, but I don't see the TZ listed.
So, my solution was to create a filter for these two hosts, create a new logging destination that points to the same destination as the regular logs (same path and file), but use the filter to split the two hosts out, apply the TZ, then dump to that file. It doesn't work. When I reload syslog-ng, it complains about the line where I define the 2nd destination pointing to the same file as the original destination.
I could create a 2nd destination with a different file, but then I'd have to configure Splunk to eat that file and make sure our log rotation scheme properly handles that file (checksums, compression, etc). This makes a simple one-off much more complex and increases the chances that someone working on this after me misses the difference between this and our other customers' devices (this is part of a managed service).
Any suggestions?
# Remote logging
source s_remote {
#tcp(ip(0.0.0.0) port(514));
udp(ip(0.0.0.0) port(514));
};
###Original Destinations###
destination d_server { udp("1.1.1.1" port(514)); };
destination d_file{
file("/output/syslog-ng/log/syslog_messages" owner("root") group ("root") perm(0640));
};
###End Original Destinations###
#### AIX Destination #### Problem here!####
#destination d_file_AIX {
# file("/output/syslog-ng/log/syslog_messages" owner("root") group ("root") perm(0640);###Note: Same path as above#
# time_zone(+1:00))
#};
#destination d_server_AIX { udp("1.1.1.1" port(514);
# time_zone(+01:00));
#};
#### END AIX Destinations ###
#### AIX Filters for proper timestamping ####
#filter AIX_Hosts { host("1") OR host("2"); };
#### END AIX Filters ###
###Original log paths###
log { source(s_remote); destination(d_server); };
log { source(s_remote); destination(d_file); };
###end original log paths###
#### AIX Logging paths###
#log { source(s_remote); filter(AIX_Hosts); destination(d_file_AIX); };
#log { source(s_remote); filter(AIX_Hosts); destination(d_server_AIX); };
#### END AIX Logging paths###
