Warning: Some posts on this platform may contain adult material intended for mature audiences only. Viewer discretion is advised. By clicking ‘Continue’, you confirm that you are 18 years or older and consent to viewing explicit content.
This is what I was thinking. And you can’t really graph out things over time on a graph which is really critical for a lot of workflows.
I get that Splunk and Elastic or unwieldy beasts that take way too much maintenance for what they provide for many orgs but to think grep is replacement is kinda crazy.
Syslog isn’t really overkill IMO. It’s pretty easy to configure it to log to a remote server, and to split particular log types or sources into different files. It’s a decent abstraction - your app that logs to syslog doesn’t have to know where the logs are going.
Since you are talking about pods, you are obviously emitting all your logs on stdout and stderr, and you have of course also labeled your pods nicely, so grepping all 36 gods is as easy as kubectl logs -l <label-key>=<label-value> | grep <search-term>
Stern has been around for ever. You could also just use a shared label selector with kubectl logs and then grep from there. You make it sound difficult if not impossible, but it’s not. Combine it with egrep and you can pretty much do anything you want right there on the CLI
I don’t know how k8s works; but if there is a way to execute just one command in a container and then exit out of it like chroot; wouldn’t it be possible to just use xargs with a list of the container names?
Good luck connecting to each of the 36 pods and grepping the file over and over again
You can run the logs command against a label so it will match all 36 pods
for X in $(seq -f host%02g 1 9); do echo $X; ssh -q $X “grep the shit”; done
:)
But yeah fair, I do actually use a big data stack for log monitoring and searching… it’s just way more usable haha
This is what I was thinking. And you can’t really graph out things over time on a graph which is really critical for a lot of workflows.
I get that Splunk and Elastic or unwieldy beasts that take way too much maintenance for what they provide for many orgs but to think grep is replacement is kinda crazy.
Let me introduce you to syslogd.
But well, it’s probably overkill, and you almost certainly just need to log on a shared volume.
Syslog isn’t really overkill IMO. It’s pretty easy to configure it to log to a remote server, and to split particular log types or sources into different files. It’s a decent abstraction - your app that logs to syslog doesn’t have to know where the logs are going.
Just write a bash script to loop over them.
Since you are talking about pods, you are obviously emitting all your logs on stdout and stderr, and you have of course also labeled your pods nicely, so grepping all 36 gods is as easy as
kubectl logs -l <label-key>=<label-value> | grep <search-term>Stern has been around for ever. You could also just use a shared label selector with
kubectl logsand then grep from there. You make it sound difficult if not impossible, but it’s not. Combine it with egrep and you can pretty much do anything you want right there on the CLIThat’s why tmux has
synchronize-panes!I don’t know how k8s works; but if there is a way to execute just one command in a container and then exit out of it like chroot; wouldn’t it be possible to just use xargs with a list of the container names?
yeah, just use kubectl and pipe stuff around with bash to make it work, pretty easy