Logging Methods

[KT Walrus]

My new site will require better logging method than just logging to the database.

It would be better if IPS implemented "Logging Methods" (with several built-in methods) like the "Storage Methods" currently support built-in and user provided options for files managed by a storage extension.

And, I'd like the option to collect more specific activity normally collected by IP in the web server logs that allows me to log all session activity to a "Logging Method". This session activity, like viewing content, or navigating to other pages would have the user ids associated with them in the logs so I can do full security audits from these logs without having to investigate the web server logs for suspicious account specific activity (the web server logs might be too hard to analyze on an account specific basis for logged in users).

[bfarber]

We used to have more robust logging options (including logging to syslog, logging to files, logging to the database, etc.) and virtually no one used them.

I think you're probably going to be best off with a plugin if this is something you need.

[KT Walrus]

20 hours ago, bfarber said:

We used to have more robust logging options (including logging to syslog, logging to files, logging to the database, etc.) and virtually no one used them.

I think that now that running in Docker/Kubernetes containers is a real thing for many of your customers, we would appreciate if you could implement a more container friendly logging method. Even if it is logging to a single file, I could always direct the file to stdout so the logs are accessible using Docker logging methods. Saving logs away in a DB is just a pain. I can implement an agent to watch a database table, but that just is extra work. 

Here is how I redirect logging for my NGINX container:

RUN mkdir -p /var/log/nginx && \
    ln -sf /dev/stdout /var/log/nginx/access.log && \
    ln -sf /dev/stderr /var/log/nginx/error.log

NGINX is configured to log to the 2 files and I simply redirect those files to stdout and stderr, respectively. If IPS4 did that, it could have an ACP setting to log session activity to access.log that is easily digested to monitor our websites. The NGINX logs are so cluttered and only track users by IP that it is much harder to analyze how users are using my site.

Good logging in IPS4 would really help most admins monitor for problems and optimize their sites for the best user experience possible.

@bfarber Hope you reconsider your opinion that your customers don't want better logging in IPS4 and we are "best off with a plugin" and not a standard supported feature.

 

[Daniel F]

You could created a minimal plugin overriding IPS\Log to utilize monolog which is really an amazing tool if you need proper logging.

 

[KT Walrus]

18 minutes ago, Daniel F said:

You could created a minimal plugin overriding IPS\Log to utilize monolog which is really an amazing tool if you need proper logging.

Good idea! I'll probably do this until IPS implements better logging through out the suite. As I wrote above, IPS4 really needs much better logging to monitor all user activity (like NGINX does for HTTP requests/responses). I really want to be able to trace all activity by all users (or a specific set of users) and not just activity that changes content. I want to be able to monitor when content is viewed as well.

So, putting logging in the controllers, nodes, active records and other core classes is probably needed too for a better logging facility. Not just redirecting logs to a file. Maybe, I'll work on hooking into these classes as well to make logging more useful to me. But, I'd rather see IPS build in better logging so I don't have to invent my own.

 

[Daniel F]

17 minutes ago, KT Walrus said:

Good idea! I'll probably do this until IPS implements better logging through out the suite. As I wrote above, IPS4 really needs much better logging to monitor all user activity (like NGINX does for HTTP requests/responses). I really want to be able to trace all activity by all users (or a specific set of users) and not just activity that changes content. I want to be able to monitor when content is viewed as well.

So, putting logging in the controllers, nodes, active records and other core classes is probably needed too for a better logging facility. Not just redirecting logs to a file. Maybe, I'll work on hooking into these classes as well to make logging more useful to me. But, I'd rather see IPS build in better logging so I don't have to invent my own.

 

I disagree with this!

There's a huge difference between logging user, moderation & admin actions and real errors & exceptions. Sure, sites with the proper infrastructure could and are probably putting everything into their logstash and elasticsearch instance (or any similar services), but average Joe doesn't have such infrastructure, and he IMO doesn't even have the need for it. It makes more sense to have a normal and separate logging and table views, with their own filtering.

[KT Walrus]

1 minute ago, Daniel F said:

I disagree with this!

Then why does NGINX separate access logging and error logging? 

It is very very useful for even the "average Joe" to be able to monitor both easily with standard monitoring services (available to all for a small monthly fee if you don't have the inclination to do this DIY).

Maybe I don't understand your reply?

5 minutes ago, Daniel F said:

It makes more sense to have a normal and separate logging and table views, with their own filtering.

I am basically suggesting this except that logging should go to files (not database) where it is more easily processed and that we should be able to log everything, only separating errors and access logs like NGINX does. The logs from an app will be much more usable than the NGINX logs since they can be much more application specific. And, built in IPS4 logging can be better curated to produce better logs than I have time to implement myself for just my site.

[Daniel F]

Sorry, yea I guess I misunderstood following part then.

Quote

So, putting logging in the controllers, nodes, active records and other core classes is probably needed too for a better logging facility.

Especially because I had the impression that you were talking about IPS\Log and not our Admin or Mod Log.

[KT Walrus]

1 minute ago, Daniel F said:

Sorry, yea I guess I misunderstood following part then.

Yes, I just thought that putting the logging into the core classes and not the app's classes is a better place to add logging. Like the DB classes (including ActiveRecords) could log all database activity (for a specific logging level where you want debug info in the logs) and at the controller level (when you want see activity logs generated) and in the Exception classes for error logging, etc.

[bfarber]

The problem is that while many admins would probably like such functionality, they also don't have the infrastructure for it. Logging every page view and every action can be quite intensive.

We have finite resources so we have to focus on things that the majority of clients may benefit from and/or that get requested a lot.

[KT Walrus]

9 hours ago, bfarber said:

We have finite resources so we have to focus on things that the majority of clients may benefit from and/or that get requested a lot.

Okay @bfarber. I trust you to focus on what is important to IPS. I don't think an activity log is all that intensive. After all, NGINX manages to log every HTTP request/response to its log files and no one complains how resource intensive that is. An IPS4 activity log would be no more server intensive that NGINX's access logs, and would be more useful to more clients than you might currently think.

In the meantime, I've implemented a simple plugin in my app to append all \IPS\Log data to a file. I persist this file on my Docker host and to view the logs cluster wide or from my macbook, I have a little container simply 'tail -f' each php server's log file. This gets the data into Docker where there are plugins for handling/monitoring the container logs.

I think it will work fine for error logging and for debug logging. And, activity logging should be easy to add if I spend a day or two to look into where to find this information perhaps session data since IPS4 updates session data with last activity info.