VCSA 6x Troubleshooting Free

From Bitbull Wiki
Jump to navigation Jump to search

General

General Documentation

Release Notes

VMware Links

LAB Links

Photon OS Administration

Console basics

  • Swtich TTY
ALT+F1 -> ... -> ALT+F12
chvt 5 # change to TTY 5
echo here I am > /dev/tty5
  • change keyboard layout on console
loadkeys us
loadkeys sg
loadkeys --help
  • redirect kernel logs to TTY
dmesg -Tw | grep -v IPfilter > /dev/tty6 & # redirect kernel logs to tty6 and put job into background
disown %1 # detacht the job above and make it save to logout
chvt 5 # switch vm console to tty5 (so humans can inspect the logs)

VIM basics

<Esc>          command mode
V              visual mode
i              insert mode
:              command line mode
:help<Enter>   find help
:r! date -R    insert date command output
qa             record into register a
q              quit recorder
@a             replay recorder with register a
:edit text.txt edit next file
:split /new    open new file in splitted screen
:wnext         save current file and jump into next one
% s/from/to/g  replace string in whole file
10g<Enter>     jump to line 10
/string        search string
:wq            save and exit
:q!            do NOT save and exit

Comandline task management

<CTRL>+<Z>  #detach job (paused)
bg          #background job (running)
jobs        #show jobs
fg          #catch job (fg)
kill -9 %1  #kill job1
disown %1   #detach job1 from tree (safe for logout)
nohup %1   #detach job1 from tree (safe for logout)

RPM basics

  • list all packages
rpm -qa
  • list last installed packages
rpm -qa --last | head
  • find rpm which is owning a file
rpm -qf /etc/dnsmasq.conf
  • check rpm for changed files
rpm -V dnsmasq
  • show all files of rpm
rpm -ql dnsmasq
  • show package info and config files, found by file
rpm -qif --configfiles /opt/vmware/etc/init.d/vami-lighttp
  • show all vmware config files, owned by rpms
rpm -qi --configfiles lighttpd
  • show all scripts, owned by rpms
rpm -qi --scripts lighttpd

tdnf basics

Photon OS Troubleshooting

Default troubleshooting tools

  • tcpdump -C 1024 -w /tmp/vcsa_dump.dat -i eth0 -s 0 host 10.77.89.34
  • netcat
  • strace -e open systemctl restart vami-lighttp
  • lsof -i -P -n | grep LISTEN
  • netstat -taupen
  • top
  • lsblk
  • blkid
  • dmesg -Tw

Logins/Reboots

  • List last Logins / Reboots
 last

Hardware

  • kernel log
dmesg -T
  • HW info
dmidecode
cat /proc/cpuinfo
  • disk information
lsblk
blkid
df -PTha
df -PThai
cat /proc/partitions

Services

  • get all systemd objects
 systemctl list-unit-files
  • get running services
systemctl --all
  • get failed services
systemctl --all --state=failed
  • useful cmds
systemctl | grep ' vm'
systemctl list-dependencies SVC_NAME
  • systemd dirs
    • /etc/systemd/system/
    • /usr/lib/systemd/system/

Logs

  • show recent changed log files
find /var/log -follow  -type f -mmin -1
tail -f /storage/log/vmware/applmgmt/monsvc.log | egrep --color=always -i '$|error|crit|warn'
lsof | egrep 'log$|out$' | awk '{print $10}' | sort -u | xargs tail -f | egrep --color=always -i '$|error|crit|warn'
lsof /var/log /storage/log | awk '{print $9}' | sort -u  | xargs tail -f | tee /tmp/all.log
journalctl -af

User / Group

  • get local user/group details
getent passwd
getent group
id $USER
chage -l $USER
  • files
/etc/passwd   # user config
/etc/shadow   # user# passwd
/etc/group    # group config
/etc/gshadow  # group password
  • unlock user account
Minimum Password Age to 0 (m)
Maximum Password Age to 99999 (M)
Password Inactive Lock after X days to -1 (I)
Account Expiration Date to -1 (E)
chage -I -1 -m 0 -M 99999 -E -1 username

Process / Performance

  • dstat live performance
dstat -cdngysl --nocolor
dstat -cdngymsp --lock --tcp
dstat -f -M time,cpu,net,disk,sys,swap,page,load,proc,topcpu --output $(date '+%Y.%m.%d-%H.%M')-dstat.csv
# convert csv2html: https://github.com/joe-speedboat/scripts/blob/master/dstat-csv2gnuplot.sh
# with vcsa 6.7, you need python2 as well: tdnf install python2 # not supported by VMw
  • Process view
top
vimtop
ps faxu | less
  • Process trace
strace -f -p SOME_PID 2>&1 | grep string
strace -e trace=open,read -p SOME_PID 
strace -c find /usr/local/ -type f 
strace -c ping -c3 -w3 8.8.8.8
strace -e trace=open,read,stat,file -p SOME_PID

Networking

  • sockets and ports and connection status
ss
netstat -taupen
lsof -i -P -n
  • tcpdump network analysis
tcpdump -C 100 -w sw1247_dump.dat -i eth0 -s 1500 host 10.77.89.34
tcpdump host client.example.com and port 80
tcpdump -x -s0 ip host vm05 and host vm06 and \( port 80 or port 8080 \)
tcpdump ip and not net 127.0.0.0
tcpdump not port 22
  • outgoing connectivity (tcp port)
nc -zv 1.2.3.4 443


Backup / Restore

  • when ever possible make snapshot of vcsa before troubleshooting
  • take care of permissions, use always cp -a

Reset Root Password

  • vcsa 6.5 trying to automate the backup - authentication failure
  • VMware RootPW recovery KB
  • Take Snapshot of the virtuell vCenter Appliance
  • Connect to ESXi Server running the vCenter Appliance
  • Connect to the vCenter Appliance Console
  • Reboot the vCenter Appliance
  • When the PhotonOS Screen is showing, press e
  • At the end of the line with beginning linux add rw init=/bin/bash
  • Press CTRL + x
  • Type passwd
  • Enter new root Password
  • umount /
  • reboot -f
hint: you an mount a ro filesystem into rw state with: mount -o remount,rw /

Unlock user after entering emergency shell

  • show user account state
chage -l root
  • unlock user
usermod -U root
  • diable password expiration
chage -M 9999 root

vCenter 6.x VCSA

vCenter 6.x Max Value difference

Type vCenter 6.0 vCenter 6.5 vCenter 6.7
OS SUSE Linux Enterprise 11 Photon OS 1.0 Photon OS 1.0
MGMT Web Client / FAT Client Web Client / HTML5 Client Web Client / HTML5 Client
max CPU/Host 480 576 768
max vCPUs/VM 128 128 128
max VMs/Host 1024 1024 1024
max vCPUs/Core 32 32 32
max vCPUs/Host 4096 4096 4096
max Mem/Host 12TB 12TB 16TB
max Hosts/Cluster 64 64 64
max VMs/Cluster 8000 8000 4000
vMEM/VM 4TB 6TB 6TB
REST API No Yes Yes

vCSA OS status archiver

vCSA Backup / Restore script

  • YOU NEED TO KEEP INSTALL MEDIA AS WELL
  • Backup
https://github.com/joe-speedboat/scripts/blob/master/vcsa_65_backup_scp.sh
https://github.com/joe-speedboat/scripts/blob/master/vcsa_67_backup_scp.sh
From within VCSA: http://www.vmwarearena.com/schedule-vcsa-6-7-backup/
  • Restore
https://github.com/joe-speedboat/scripts/blob/master/vcsa_65_restore_scp.sh
https://github.com/joe-speedboat/scripts/blob/master/vcsa_67_restore_scp.sh
Debug Log: /storage/log/vmware/applmgmt/backupRestoreAPI.log
WARNING: PNID and system.name must be equal to run restore

[root@deployment vcenter]# grep vcenter1 /media/vmware/vcenter/json/vcenter1.json
            "name": "vcenter1"
            "system.name": "vcenter1.lab.local"

backup-host:~/vcenter1.lab.local/2018-04-24-14-07-35 # grep vcenter1.lab.local backup-metadata.json 
    "PNID": "vcenter1.lab.local",

Service Control

  • list services
service-control --list
  • check status of services
service-control --list | cut -d' ' -f1 | while read svc
do 
   systemctl status $svc
   echo "--------------------------------"
done
Photon OS Commands SLES Commands
systemctl start <service> service <service> start
systemctl stop <service> service <service> stop
systemctl restart <service> service <service> restart
systemctl reload <service> service <service> reload
systemctl condrestart <service> service <service> condrestart
systemctl status <service> service <service> status
systemctl enable <service> chkconfig <service> on
systemctl disable <service> chkconfig <service> off
systemctl daemon-reload chkconfig <service> --add
systemctl list-units --type service --all chkconfig --list <service name>
systemctl is-enabled <service> chkconfig --list vmware-tools-services

VCSA Services

Service Name Description
applmgmt VMware Appliance Management Service
vmware-cis-license VMware License Service
vmware-cm VMware Component Manager
vmware-eam VMware ESX Agent Manager
vmware-sts-idmd VMware Identity Management Service
vmware-invsvc VMware Inventory Service
vmware-mbcs VMware Message Bus Configuration Service
vmware-netdumper VMware vSphere ESXi Dump Collector
vmware-perfcharts VMware Performance Charts
vmware-rbd-watchdog VMware vSphere Auto Deploy Waiter
vmware-rhttpproxy VMware HTTP Reverse Proxy
vmware-sca VMware Service Control Agent
vmware-sps VMware vSphere Profile-Driven Storage Service
vmware-stsd VMware Security Token Service
vmware-syslog VMware Common Logging Service
vmware-syslog-health VMware Syslog Health Service
vmware-vapi-endpoint VMware vAPI Endpoint
vmware-vdcs VMware Content Library Service
vmafdd VMware Authentication Framework
vmcad VMware Certificate Service
vmdird VMware Directory Service
vmware-vpostgres VMware Postgres
vmware-vpx-workflow VMware vCenter Workflow Manager
vmware-vpxd VMware vCenter Server
vmware-vsm VMware vService Manager
vsphere-client vSphere Web Client
vmware-vws VMware System and Hardware Health Manager
vmware-vsan-health VMware vSAN Health Service

Enable SSH

  • WebUI
https://vcenter/vsphere-client
Administration > System Configuration > Nodes > vCenter > Manage > Settings > Access
  • VMI
https://vcenter:5480
Access > SSH

Modify vCSA Iptables rules

  • this is not perisistent across reboots!
iptables-save > iptables.rules # save current rules to file
iptables-save | grep 443       # find rules for https
iptables -D port_filter -p tcp -m tcp --dport 443 -j ACCEPT   # delete https input rule (replace -A with -D)
ip a                           # find your ip
iptables -A port_filter -p tcp -m tcp -s 192.168.3.220 --dport 443 -j ACCEPT   # inject custom rule with your ip
# now only your ip can access https interface (be aware of APIs which may also fail)
iptables-restore < iptables.rules  # restore old rules when work done

Change root shell

WinSCP transfer fails

NOTE: with bash shell, you wont hit this issue

  • Log in vCSA VAMI UI (https://vcsa-ip:5480)
  • Under Access, enable SSH Login. (PS: enabling Bash Shell is not necessary)
  • Open WinSCP, select File protocol: SFTP
  • Enter the vCSA hostname, port number 22, root, and root’s password
    • Click Advanced
    • Under Environment, SFTP, Protocol options
    • Set SFTP server to shell /usr/libexec/sftp-server (vcsa 6.5)
    • Set SFTP server to shell /usr/libexec/sftp-server (vcsa 6.7)
    • Set SFTP server to shell /usr/lib64/ssh/sftp-server (vcsa 6.0)

replace certificate on vcsa

SSL/TLS debugging

  • Get all LISTEN Ports and test for Certificate details by web request
IF=$(ip r | grep default | awk '{print $5}')
IP=$(ip addr show $IF | grep "inet " | awk '{print $2}' | cut -d'/' -f1)
lsof -i -P -n 2>/dev/null | grep LISTEN | grep -v 127.0.0.1 | awk '{print $9}' | cut -d: -f2 | sort -u | while read port 
do 
   echo "---------- https://$IP:$port "
   lsof -i -P -n 2>/dev/null | grep LISTEN | grep :$port 
   curl -m3 -k -vv https://$IP:$port 2>&1 | grep -A4 'Server certificate:'
done

Upgrade VCSA 6.0 to 6.5

VAMI comandline access

  • * WARNING: you easily f... .p the VCSA with this commands
  • show all the VMI cli comands
ls -1 --color=never /opt/vmware/share/vami/vami_*
  • some useful read cmds
/opt/vmware/share/vami/vami_dns
/opt/vmware/share/vami/vami_get_network eth0
/opt/vmware/share/vami/vami_set_proxy
/opt/vmware/share/vami/vami_set_timezone

Appliance Shell examples

  • this is the recommended way to change things inside vcsa
Command> help api list
Command> ntp.get
Command> ntp.server.set --help
  • Script your config
vim vcsa_api.cmd 
------
dns.domains.set  --domains office.bitbull.ch
ntp.set --servers pool.ntp.org
dns.domains.list
ntp.get
------

cat vcsa_api.cmd | ssh root@vc05 
   Pseudo-terminal will not be allocated because stdin is not a terminal.
   VMware vCenter Server Appliance 6.7.0.30000

   root@vc05's password: 
   Connected to service

    * List APIs: "help api list"
    * List Plugins: "help pi list"
    * Launch BASH: "shell"

   Command> Command> Command> Domains: office.bitbull.ch
   Command> Servers: pool.ntp.org

Install ssh public key manually

  • Client host, create key pair
ssh-keygen
cat .ssh/id_???.pub
# copy rsa key to clipboard
  • on target system
cd /root
mkdir .ssh
chmod 700 .ssh
cd .ssh
touch authorized_keys
chmod 600 authorized_keys
echo 'my-super-public-key' >> /root/.ssh/authorized_keys
  • keep in mind
it only works with bash as shell and not with appliance shell from vmware
https://kb.vmware.com/s/article/2146205

Troubleshooting full filesystem

Find big files

root@vc04 [ /storage/log ]# df -Th | grep ext
     /dev/sda3                                ext4       11G  5.1G  5.0G  51% /
     /dev/mapper/log_vg-log                   ext4      9.8G  9.8G    0G 100% /storage/log
     /dev/mapper/updatemgr_vg-updatemgr       ext4       99G   82M   94G   1% /storage/updatemgr
     /dev/mapper/autodeploy_vg-autodeploy     ext4      9.8G   23M  9.2G   1% /storage/autodeploy
     /dev/sda1                                ext4      120M   34M   78M  31% /boot
     /dev/mapper/netdump_vg-netdump           ext4      985M  1.3M  916M   1% /storage/netdump
     /dev/mapper/core_vg-core                 ext4       25G   88M   24G   1% /storage/core
     /dev/mapper/imagebuilder_vg-imagebuilder ext4      9.8G   23M  9.2G   1% /storage/imagebuilder
     /dev/mapper/db_vg-db                     ext4      9.8G  102M  9.1G   2% /storage/db
     /dev/mapper/seat_vg-seat                 ext4      9.8G   55M  9.2G   1% /storage/seat
     /dev/mapper/archive_vg-archive           ext4       50G   64M   47G   1% /storage/archive
     /dev/mapper/dblog_vg-dblog               ext4       15G  102M   14G   1% /storage/dblog


root@vc04 [ /storage/log ]# cd /storage/log/vmware
root@vc04 [ /storage/log ]# du -mcs * | sort -n | tail 
     1	lost+found
     1	remote
     1	vcsa_67_status-week-20.log
     1	vcsa_67_status-week-21.log
     9905	vmware
     9905	total

root@vc04 [ /storage/log/vmware ]# du -mcs * | sort -n | tail 
     3	vsphere-ui
     4	vapi
     5	procstate-20190521.gz
     7	vpxd
     7	vsphere-client
     9	vsan-health
     12	sso
     41	procstate
     9807	cis-license
     9905	total

root@vc04 [ /storage/log/vmware/cis-license ]# du -mcs * | sort -n | tail
     1	license.log.1.gz
     1	license.log.2.gz
     1	license-runtime.log-0.stderr
     1	license-runtime.log-0.stdout
     1	license-runtime.log-1.stderr
     1	license-runtime.log-1.stdout
     1	license-runtime.log.stderr
     1	license-runtime.log.stdout
     9807	total
     9807	vmware-cis-license-gc.log.0.current

# File found. Is it safe to delete it?
root@vc04 [ /storage/log/vmware/cis-license ]# lsof vmware-cis-license-gc.log.0.current
     COMMAND    PID USER   FD   TYPE DEVICE    SIZE/OFF   NODE NAME
     vmware-ci 2095 root    3w   REG  252,6 10282348544 524436 vmware-cis-license-gc.log.0.current

truncate big file which is in use

  • Oh no, file is open if we delete, process will probably freak out
so we truncate the file instead of deleting it
# File found. Is it safe to delete it?
root@vc04 [ /storage/log/vmware/cis-license ]# lsof vmware-cis-license-gc.log.0.current
     COMMAND    PID USER   FD   TYPE DEVICE    SIZE/OFF   NODE NAME
     vmware-ci 2095 root    3w   REG  252,6 10282348544 524436 vmware-cis-license-gc.log.0.current

root@vc04 [ /storage/log/vmware/cis-license ]# ls -lahi vmware-cis-license-gc.log.0.current
     524453 -rw-r--r-- 1 root root 9.7G May 21 11:39 vmware-cis-license-gc.log.0.current

root@vc04 [ /storage/log/vmware/cis-license ]# > vmware-cis-license-gc.log.0.current

root@vc04 [ /storage/log/vmware/cis-license ]# ls -lahi vmware-cis-license-gc.log.0.current
     524453 -rw-r--r-- 1 root root 0 May 21 11:41 vmware-cis-license-gc.log.0.current

root@vc04 [ /storage/log/vmware/cis-license ]# df -hT .
     Filesystem             Type  Size  Used Avail Use% Mounted on
     /dev/mapper/log_vg-log ext4  9.8G   54M  9.2G   1% /storage/log

delete logfiles which are not in use (should not be necessary)

  • Try first: manually kick log rotation (new in v6.7)
root@vc05 [ /etc/cron.d ]# ls -lah /etc/cron.d/*log*
     -rw-r--r-- 1 root root 62 Mar 27 05:30 /etc/cron.d/logdiskcheck.cron
     -rw-r--r-- 1 root root 97 May 20 13:21 /etc/cron.d/logrotate.cron
root@vc05 [ /etc/cron.d ]# cat /etc/cron.d/logdiskcheck.cron
     */10 * * * *   root /usr/sbin/logdiskcheck.sh >/dev/null 2>&1

root@vc05 [ /etc/cron.d ]# cat /etc/cron.d/logrotate.cron
     0 */1 * * *   root  /usr/sbin/logrotate /etc/logrotate.conf 2>&1 |  /usr/bin/logger -t logrotate

root@vc05 [ /etc/cron.d ]# cat /usr/sbin/logdiskcheck.sh
     log_dir="/var/log/vmware/"
     max_usage_percent=90
     usage_percent=$(df $log_dir | tail -1 | awk '{print $5+0}')
     if [ $usage_percent -lt $max_usage_percent ]; then
         exit 0
     fi
     /usr/bin/logger -t logdiskcheck "WARN logdisk nearly full. Attempt to cleanup."
     # Force a log rotate.
     /usr/sbin/logrotate -f /etc/logrotate.conf
     # Delete more than a week old hprof files from component log directory.
     find "$log_dir" -mindepth 2 -mtime +7 -type f -name '*.hprof' -delete

root@vc05 [ /etc/cron.d ]# sh -x /usr/sbin/logdiskcheck.sh
      + log_dir=/var/log/vmware/
      + max_usage_percent=90
      ++ df /var/log/vmware/
      ++ awk '{print $5+0}'
      ++ tail -1
      + usage_percent=100
      + '[' 100 -lt 90 ']'
      + /usr/bin/logger -t logdiskcheck 'WARN logdisk nearly full. Attempt to cleanup.'
      + /usr/sbin/logrotate -f /etc/logrotate.conf
      error: error creating output file /var/log/vmware/vpostgres/pg_archiver.log.stderr.1: File exists
      error: error creating output file /var/log/vmware/vpostgres/pg_archiver.log.stdout.1: File exists
      error: error creating output file /var/log/vmware/vpostgres/postgres_cron.log.1: File exists
      + find /var/log/vmware/ -mindepth 2 -mtime +7 -type f -name '*.hprof' -delete
root@vc05 [ /etc/cron.d ]# df /storage/log
      Filesystem             1K-blocks     Used Available Use% Mounted on
      /dev/mapper/log_vg-log  10181944 10164068         0 100% /storage/log

This is NORMALLY not needed, VMware manually kicks logrotation every 10min if filesystem /storage/log/vmware is filled more than 90% BUT BE AWARE: not all log files are maintained by vmware config, ours is not!!!

delete logfiles manually (last option)

  • big file is not used by any process (lsof big-file)
  • logrotate script (step above) did not cleanup the disk filling problem
root@vc05 [ /storage/log/vmware/cis-license ]# ls -lah vmware-cis-license-gc.log.1
     -rw-r--r-- 1 root root 9.7G May 21 12:39 vmware-cis-license-gc.log.1

root@vc05 [ /storage/log/vmware/cis-license ]# lsof vmware-cis-license-gc.log.1 #file must not be in use

# truncating log is better then deleting it. logrotate may expect proper file numbering
root@vc05 [ /storage/log/vmware/cis-license ]# > vmware-cis-license-gc.log.1

root@vc05 [ /storage/log/vmware/cis-license ]# ls -lah vmware-cis-license-gc.log.1
     -rw-r--r-- 1 root root 0 May 21 12:53 vmware-cis-license-gc.log.1

root@vc05 [ /storage/log/vmware/cis-license ]# df -hT .
     Filesystem             Type  Size  Used Avail Use% Mounted on
     /dev/mapper/log_vg-log ext4  9.8G   58M  9.2G   1% /storage/log

Access/Troubleshoot PostgreSQL DB

  • fetch credentials as root user
cat /etc/vmware-vpx/embedded_db.cfg
  • swith into disabled vpostgres user
su -s /bin/bash vpostgres
  • connect database
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

log purge on vCSA 6.x

https://kb.vmware.com/s/article/2110031
curl -s "https://kb.vmware.com/servlet/fileField?entityId=ka234000000TvzOAAS&field=Attachment_3__Body__s" > 2110031_Postgres_task_event_stat.sql

curl -s https://raw.githubusercontent.com/lamw/vghetto-scripts/master/shell/queryVCDBvPostgres.sh | sh

/opt/vmware/vpostgres/current/bin/psql -U postgres -v TaskMaxAgeInDays=7 -v EventMaxAgeInDays=7 -v StatMaxAgeInDays=7 -d VCDB -t -q -f 2110031_Postgres_task_event_stat.sql

curl -s https://raw.githubusercontent.com/lamw/vghetto-scripts/master/shell/queryVCDBvPostgres.sh | sh

purge vcsa logs by truncating (not verified by VMware)

  • If db is full then regular kb will lead into error because temporary space is needed but not available
  • Find an idea what to do and how others do get around the problem:
https://communities.vmware.com/thread/560901
https://kb.vmware.com/s/article/1025914
  • get an idea on what to delete
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c '\dt
  • prepare truncate psql statement as you need copy paste into sql prompt:
[chris@lts1 ~]$ seq 100 | while read nr; do echo "truncate table vpx_event_$nr cascade;"; done
truncate table vpx_event_1 cascade;
[...]
truncate table vpx_event_100 cascade;
  • Open SQL Prompt and paste the comand from step above


  • Example db login:
http://www.postgresqltutorial.com/postgresql-cheat-sheet/
 /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
    psql.bin (9.4.9 (VMware Postgres 9.4.9.0-4442725 release))
    Type "help" for help.
    VCDB=#
        SELECT * FROM VPX_VERSION;
        SELECT * FROM VPX_HOST;
        \q

vCSA troubleshooting KBs

vCSA Important KBs

VCSA KBs to review

vCSA KBs to discuss

More fun

ESXi console

OpenSSL comandline

LAB exercises to discuss

mount | column -t

ls -lR / >std_out.log 2>err.log
cat std_out.log 
cat err.log

lsof -i -P -n | grep LISTEN | egrep -v '::1|127.0.0.1' | awk '{print $1 " "$9}' | sort -u

find /var/log /storage/log -follow  -type f -mmin -5

find /var/log /storage/log -follow  -type f -mmin -5 | xargs file | grep ASCII | cut -d: -f1 | xargs tail -f
lsof | egrep 'log$|out$' | awk '{print $10}' | sort -u | xargs tail -f | egrep --color=always -i '$|error|crit|warn'

lsof /var/log /storage/log | awk '{print $9}' | sort -u  | xargs tail -f | tee -a /tmp/all.log

reset

curl ifconfig.me

lsof /storage/log/vmware/vapi/endpoint/*



# log your work
some-important-cmd | tee -a /root/debug.log
script -a /root/my_session.script
Retrieved from "http://www.bitbull.ch/wiki/index.php?title=VCSA_6x_Troubleshooting_Free&oldid=5655"