VCSA 6x Troubleshooting Free

From Bitbull Wiki
Revision as of 12:27, 5 April 2023 by Chris (talk | contribs) (Chris moved page Protected VCSA 6x Troubleshooting to VCSA 6x Troubleshooting Free)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

1 General

1.1 General Documentation

1.2 Release Notes

1.3 VMware Links

1.4 LAB Links

2 Photon OS Administration

2.1 Console basics

  • Swtich TTY
ALT+F1 -> ... -> ALT+F12
chvt 5 # change to TTY 5
echo here I am > /dev/tty5
  • change keyboard layout on console
loadkeys us
loadkeys sg
loadkeys --help
  • redirect kernel logs to TTY
dmesg -Tw | grep -v IPfilter > /dev/tty6 & # redirect kernel logs to tty6 and put job into background
disown %1 # detacht the job above and make it save to logout
chvt 5 # switch vm console to tty5 (so humans can inspect the logs)

2.2 VIM basics

<Esc>          command mode
V              visual mode
i              insert mode
:              command line mode
:help<Enter>   find help
:r! date -R    insert date command output
qa             record into register a
q              quit recorder
@a             replay recorder with register a
:edit text.txt edit next file
:split /new    open new file in splitted screen
:wnext         save current file and jump into next one
% s/from/to/g  replace string in whole file
10g<Enter>     jump to line 10
/string        search string
:wq            save and exit
:q!            do NOT save and exit

2.3 Comandline task management

<CTRL>+<Z>  #detach job (paused)
bg          #background job (running)
jobs        #show jobs
fg          #catch job (fg)
kill -9 %1  #kill job1
disown %1   #detach job1 from tree (safe for logout)
nohup %1   #detach job1 from tree (safe for logout)

2.4 RPM basics

  • list all packages
rpm -qa
  • list last installed packages
rpm -qa --last | head
  • find rpm which is owning a file
rpm -qf /etc/dnsmasq.conf
  • check rpm for changed files
rpm -V dnsmasq
  • show all files of rpm
rpm -ql dnsmasq
  • show package info and config files, found by file
rpm -qif --configfiles /opt/vmware/etc/init.d/vami-lighttp
  • show all vmware config files, owned by rpms
rpm -qi --configfiles lighttpd
  • show all scripts, owned by rpms
rpm -qi --scripts lighttpd

3 tdnf basics

4 Photon OS Troubleshooting

4.1 Default troubleshooting tools

  • tcpdump -C 1024 -w /tmp/vcsa_dump.dat -i eth0 -s 0 host 10.77.89.34
  • netcat
  • strace -e open systemctl restart vami-lighttp
  • lsof -i -P -n | grep LISTEN
  • netstat -taupen
  • top
  • lsblk
  • blkid
  • dmesg -Tw

4.2 Logins/Reboots

  • List last Logins / Reboots
 last

4.3 Hardware

  • kernel log
dmesg -T
  • HW info
dmidecode
cat /proc/cpuinfo
  • disk information
lsblk
blkid
df -PTha
df -PThai
cat /proc/partitions

4.4 Services

  • get all systemd objects
 systemctl list-unit-files
  • get running services
systemctl --all
  • get failed services
systemctl --all --state=failed
  • useful cmds
systemctl | grep ' vm'
systemctl list-dependencies SVC_NAME
  • systemd dirs
    • /etc/systemd/system/
    • /usr/lib/systemd/system/

4.5 Logs

  • show recent changed log files
find /var/log -follow  -type f -mmin -1
tail -f /storage/log/vmware/applmgmt/monsvc.log | egrep --color=always -i '$|error|crit|warn'
lsof | egrep 'log$|out$' | awk '{print $10}' | sort -u | xargs tail -f | egrep --color=always -i '$|error|crit|warn'
lsof /var/log /storage/log | awk '{print $9}' | sort -u  | xargs tail -f | tee /tmp/all.log
journalctl -af

4.6 User / Group

  • get local user/group details
getent passwd
getent group
id $USER
chage -l $USER
  • files
/etc/passwd   # user config
/etc/shadow   # user# passwd
/etc/group    # group config
/etc/gshadow  # group password
  • unlock user account
Minimum Password Age to 0 (m)
Maximum Password Age to 99999 (M)
Password Inactive Lock after X days to -1 (I)
Account Expiration Date to -1 (E)
chage -I -1 -m 0 -M 99999 -E -1 username

4.7 Process / Performance

  • dstat live performance
dstat -cdngysl --nocolor
dstat -cdngymsp --lock --tcp
dstat -f -M time,cpu,net,disk,sys,swap,page,load,proc,topcpu --output $(date '+%Y.%m.%d-%H.%M')-dstat.csv
# convert csv2html: https://github.com/joe-speedboat/scripts/blob/master/dstat-csv2gnuplot.sh
# with vcsa 6.7, you need python2 as well: tdnf install python2 # not supported by VMw
  • Process view
top
vimtop
ps faxu | less
  • Process trace
strace -f -p SOME_PID 2>&1 | grep string
strace -e trace=open,read -p SOME_PID 
strace -c find /usr/local/ -type f 
strace -c ping -c3 -w3 8.8.8.8
strace -e trace=open,read,stat,file -p SOME_PID

4.8 Networking

  • sockets and ports and connection status
ss
netstat -taupen
lsof -i -P -n
  • tcpdump network analysis
tcpdump -C 100 -w sw1247_dump.dat -i eth0 -s 1500 host 10.77.89.34
tcpdump host client.example.com and port 80
tcpdump -x -s0 ip host vm05 and host vm06 and \( port 80 or port 8080 \)
tcpdump ip and not net 127.0.0.0
tcpdump not port 22
  • outgoing connectivity (tcp port)
nc -zv 1.2.3.4 443


4.9 Backup / Restore

  • when ever possible make snapshot of vcsa before troubleshooting
  • take care of permissions, use always cp -a

4.10 Reset Root Password

  • vcsa 6.5 trying to automate the backup - authentication failure
  • VMware RootPW recovery KB
  • Take Snapshot of the virtuell vCenter Appliance
  • Connect to ESXi Server running the vCenter Appliance
  • Connect to the vCenter Appliance Console
  • Reboot the vCenter Appliance
  • When the PhotonOS Screen is showing, press e
  • At the end of the line with beginning linux add rw init=/bin/bash
  • Press CTRL + x
  • Type passwd
  • Enter new root Password
  • umount /
  • reboot -f
hint: you an mount a ro filesystem into rw state with: mount -o remount,rw /

4.11 Unlock user after entering emergency shell

  • show user account state
chage -l root
  • unlock user
usermod -U root
  • diable password expiration
chage -M 9999 root

5 vCenter 6.x VCSA

5.1 vCenter 6.x Max Value difference

Type vCenter 6.0 vCenter 6.5 vCenter 6.7
OS SUSE Linux Enterprise 11 Photon OS 1.0 Photon OS 1.0
MGMT Web Client / FAT Client Web Client / HTML5 Client Web Client / HTML5 Client
max CPU/Host 480 576 768
max vCPUs/VM 128 128 128
max VMs/Host 1024 1024 1024
max vCPUs/Core 32 32 32
max vCPUs/Host 4096 4096 4096
max Mem/Host 12TB 12TB 16TB
max Hosts/Cluster 64 64 64
max VMs/Cluster 8000 8000 4000
vMEM/VM 4TB 6TB 6TB
REST API No Yes Yes

5.2 vCSA OS status archiver

5.3 vCSA Backup / Restore script

  • YOU NEED TO KEEP INSTALL MEDIA AS WELL
  • Backup
https://github.com/joe-speedboat/scripts/blob/master/vcsa_65_backup_scp.sh
https://github.com/joe-speedboat/scripts/blob/master/vcsa_67_backup_scp.sh
From within VCSA: http://www.vmwarearena.com/schedule-vcsa-6-7-backup/
  • Restore
https://github.com/joe-speedboat/scripts/blob/master/vcsa_65_restore_scp.sh
https://github.com/joe-speedboat/scripts/blob/master/vcsa_67_restore_scp.sh
Debug Log: /storage/log/vmware/applmgmt/backupRestoreAPI.log
WARNING: PNID and system.name must be equal to run restore

[root@deployment vcenter]# grep vcenter1 /media/vmware/vcenter/json/vcenter1.json
            "name": "vcenter1"
            "system.name": "vcenter1.lab.local"

backup-host:~/vcenter1.lab.local/2018-04-24-14-07-35 # grep vcenter1.lab.local backup-metadata.json 
    "PNID": "vcenter1.lab.local",

5.4 Service Control

  • list services
service-control --list
  • check status of services
service-control --list | cut -d' ' -f1 | while read svc
do 
   systemctl status $svc
   echo "--------------------------------"
done
Photon OS Commands SLES Commands
systemctl start <service> service <service> start
systemctl stop <service> service <service> stop
systemctl restart <service> service <service> restart
systemctl reload <service> service <service> reload
systemctl condrestart <service> service <service> condrestart
systemctl status <service> service <service> status
systemctl enable <service> chkconfig <service> on
systemctl disable <service> chkconfig <service> off
systemctl daemon-reload chkconfig <service> --add
systemctl list-units --type service --all chkconfig --list <service name>
systemctl is-enabled <service> chkconfig --list vmware-tools-services

5.5 VCSA Services

Service Name Description
applmgmt VMware Appliance Management Service
vmware-cis-license VMware License Service
vmware-cm VMware Component Manager
vmware-eam VMware ESX Agent Manager
vmware-sts-idmd VMware Identity Management Service
vmware-invsvc VMware Inventory Service
vmware-mbcs VMware Message Bus Configuration Service
vmware-netdumper VMware vSphere ESXi Dump Collector
vmware-perfcharts VMware Performance Charts
vmware-rbd-watchdog VMware vSphere Auto Deploy Waiter
vmware-rhttpproxy VMware HTTP Reverse Proxy
vmware-sca VMware Service Control Agent
vmware-sps VMware vSphere Profile-Driven Storage Service
vmware-stsd VMware Security Token Service
vmware-syslog VMware Common Logging Service
vmware-syslog-health VMware Syslog Health Service
vmware-vapi-endpoint VMware vAPI Endpoint
vmware-vdcs VMware Content Library Service
vmafdd VMware Authentication Framework
vmcad VMware Certificate Service
vmdird VMware Directory Service
vmware-vpostgres VMware Postgres
vmware-vpx-workflow VMware vCenter Workflow Manager
vmware-vpxd VMware vCenter Server
vmware-vsm VMware vService Manager
vsphere-client vSphere Web Client
vmware-vws VMware System and Hardware Health Manager
vmware-vsan-health VMware vSAN Health Service

5.6 Enable SSH

  • WebUI
https://vcenter/vsphere-client
Administration > System Configuration > Nodes > vCenter > Manage > Settings > Access
  • VMI
https://vcenter:5480
Access > SSH

5.7 Modify vCSA Iptables rules

  • this is not perisistent across reboots!
iptables-save > iptables.rules # save current rules to file
iptables-save | grep 443       # find rules for https
iptables -D port_filter -p tcp -m tcp --dport 443 -j ACCEPT   # delete https input rule (replace -A with -D)
ip a                           # find your ip
iptables -A port_filter -p tcp -m tcp -s 192.168.3.220 --dport 443 -j ACCEPT   # inject custom rule with your ip
# now only your ip can access https interface (be aware of APIs which may also fail)
iptables-restore < iptables.rules  # restore old rules when work done

5.8 Change root shell

5.9 WinSCP transfer fails

NOTE: with bash shell, you wont hit this issue

  • Log in vCSA VAMI UI (https://vcsa-ip:5480)
  • Under Access, enable SSH Login. (PS: enabling Bash Shell is not necessary)
  • Open WinSCP, select File protocol: SFTP
  • Enter the vCSA hostname, port number 22, root, and root’s password
    • Click Advanced
    • Under Environment, SFTP, Protocol options
    • Set SFTP server to shell /usr/libexec/sftp-server (vcsa 6.5)
    • Set SFTP server to shell /usr/libexec/sftp-server (vcsa 6.7)
    • Set SFTP server to shell /usr/lib64/ssh/sftp-server (vcsa 6.0)

5.10 replace certificate on vcsa

5.11 SSL/TLS debugging

  • Get all LISTEN Ports and test for Certificate details by web request
IF=$(ip r | grep default | awk '{print $5}')
IP=$(ip addr show $IF | grep "inet " | awk '{print $2}' | cut -d'/' -f1)
lsof -i -P -n 2>/dev/null | grep LISTEN | grep -v 127.0.0.1 | awk '{print $9}' | cut -d: -f2 | sort -u | while read port 
do 
   echo "---------- https://$IP:$port "
   lsof -i -P -n 2>/dev/null | grep LISTEN | grep :$port 
   curl -m3 -k -vv https://$IP:$port 2>&1 | grep -A4 'Server certificate:'
done

5.12 Upgrade VCSA 6.0 to 6.5

5.13 VAMI comandline access

  • * WARNING: you easily f... .p the VCSA with this commands
  • show all the VMI cli comands
ls -1 --color=never /opt/vmware/share/vami/vami_*
  • some useful read cmds
/opt/vmware/share/vami/vami_dns
/opt/vmware/share/vami/vami_get_network eth0
/opt/vmware/share/vami/vami_set_proxy
/opt/vmware/share/vami/vami_set_timezone

5.14 Appliance Shell examples

  • this is the recommended way to change things inside vcsa
Command> help api list
Command> ntp.get
Command> ntp.server.set --help
  • Script your config
vim vcsa_api.cmd 
------
dns.domains.set  --domains office.bitbull.ch
ntp.set --servers pool.ntp.org
dns.domains.list
ntp.get
------

cat vcsa_api.cmd | ssh root@vc05 
   Pseudo-terminal will not be allocated because stdin is not a terminal.
   VMware vCenter Server Appliance 6.7.0.30000

   root@vc05's password: 
   Connected to service

    * List APIs: "help api list"
    * List Plugins: "help pi list"
    * Launch BASH: "shell"

   Command> Command> Command> Domains: office.bitbull.ch
   Command> Servers: pool.ntp.org

5.15 Install ssh public key manually

  • Client host, create key pair
ssh-keygen
cat .ssh/id_???.pub
# copy rsa key to clipboard
  • on target system
cd /root
mkdir .ssh
chmod 700 .ssh
cd .ssh
touch authorized_keys
chmod 600 authorized_keys
echo 'my-super-public-key' >> /root/.ssh/authorized_keys
  • keep in mind
it only works with bash as shell and not with appliance shell from vmware
https://kb.vmware.com/s/article/2146205

5.16 Troubleshooting full filesystem

5.16.1 Find big files

root@vc04 [ /storage/log ]# df -Th | grep ext
     /dev/sda3                                ext4       11G  5.1G  5.0G  51% /
     /dev/mapper/log_vg-log                   ext4      9.8G  9.8G    0G 100% /storage/log
     /dev/mapper/updatemgr_vg-updatemgr       ext4       99G   82M   94G   1% /storage/updatemgr
     /dev/mapper/autodeploy_vg-autodeploy     ext4      9.8G   23M  9.2G   1% /storage/autodeploy
     /dev/sda1                                ext4      120M   34M   78M  31% /boot
     /dev/mapper/netdump_vg-netdump           ext4      985M  1.3M  916M   1% /storage/netdump
     /dev/mapper/core_vg-core                 ext4       25G   88M   24G   1% /storage/core
     /dev/mapper/imagebuilder_vg-imagebuilder ext4      9.8G   23M  9.2G   1% /storage/imagebuilder
     /dev/mapper/db_vg-db                     ext4      9.8G  102M  9.1G   2% /storage/db
     /dev/mapper/seat_vg-seat                 ext4      9.8G   55M  9.2G   1% /storage/seat
     /dev/mapper/archive_vg-archive           ext4       50G   64M   47G   1% /storage/archive
     /dev/mapper/dblog_vg-dblog               ext4       15G  102M   14G   1% /storage/dblog


root@vc04 [ /storage/log ]# cd /storage/log/vmware
root@vc04 [ /storage/log ]# du -mcs * | sort -n | tail 
     1	lost+found
     1	remote
     1	vcsa_67_status-week-20.log
     1	vcsa_67_status-week-21.log
     9905	vmware
     9905	total

root@vc04 [ /storage/log/vmware ]# du -mcs * | sort -n | tail 
     3	vsphere-ui
     4	vapi
     5	procstate-20190521.gz
     7	vpxd
     7	vsphere-client
     9	vsan-health
     12	sso
     41	procstate
     9807	cis-license
     9905	total

root@vc04 [ /storage/log/vmware/cis-license ]# du -mcs * | sort -n | tail
     1	license.log.1.gz
     1	license.log.2.gz
     1	license-runtime.log-0.stderr
     1	license-runtime.log-0.stdout
     1	license-runtime.log-1.stderr
     1	license-runtime.log-1.stdout
     1	license-runtime.log.stderr
     1	license-runtime.log.stdout
     9807	total
     9807	vmware-cis-license-gc.log.0.current

# File found. Is it safe to delete it?
root@vc04 [ /storage/log/vmware/cis-license ]# lsof vmware-cis-license-gc.log.0.current
     COMMAND    PID USER   FD   TYPE DEVICE    SIZE/OFF   NODE NAME
     vmware-ci 2095 root    3w   REG  252,6 10282348544 524436 vmware-cis-license-gc.log.0.current

5.16.2 truncate big file which is in use

  • Oh no, file is open if we delete, process will probably freak out
so we truncate the file instead of deleting it
# File found. Is it safe to delete it?
root@vc04 [ /storage/log/vmware/cis-license ]# lsof vmware-cis-license-gc.log.0.current
     COMMAND    PID USER   FD   TYPE DEVICE    SIZE/OFF   NODE NAME
     vmware-ci 2095 root    3w   REG  252,6 10282348544 524436 vmware-cis-license-gc.log.0.current

root@vc04 [ /storage/log/vmware/cis-license ]# ls -lahi vmware-cis-license-gc.log.0.current
     524453 -rw-r--r-- 1 root root 9.7G May 21 11:39 vmware-cis-license-gc.log.0.current

root@vc04 [ /storage/log/vmware/cis-license ]# > vmware-cis-license-gc.log.0.current

root@vc04 [ /storage/log/vmware/cis-license ]# ls -lahi vmware-cis-license-gc.log.0.current
     524453 -rw-r--r-- 1 root root 0 May 21 11:41 vmware-cis-license-gc.log.0.current

root@vc04 [ /storage/log/vmware/cis-license ]# df -hT .
     Filesystem             Type  Size  Used Avail Use% Mounted on
     /dev/mapper/log_vg-log ext4  9.8G   54M  9.2G   1% /storage/log

5.16.3 delete logfiles which are not in use (should not be necessary)

  • Try first: manually kick log rotation (new in v6.7)
root@vc05 [ /etc/cron.d ]# ls -lah /etc/cron.d/*log*
     -rw-r--r-- 1 root root 62 Mar 27 05:30 /etc/cron.d/logdiskcheck.cron
     -rw-r--r-- 1 root root 97 May 20 13:21 /etc/cron.d/logrotate.cron
root@vc05 [ /etc/cron.d ]# cat /etc/cron.d/logdiskcheck.cron
     */10 * * * *   root /usr/sbin/logdiskcheck.sh >/dev/null 2>&1

root@vc05 [ /etc/cron.d ]# cat /etc/cron.d/logrotate.cron
     0 */1 * * *   root  /usr/sbin/logrotate /etc/logrotate.conf 2>&1 |  /usr/bin/logger -t logrotate

root@vc05 [ /etc/cron.d ]# cat /usr/sbin/logdiskcheck.sh
     log_dir="/var/log/vmware/"
     max_usage_percent=90
     usage_percent=$(df $log_dir | tail -1 | awk '{print $5+0}')
     if [ $usage_percent -lt $max_usage_percent ]; then
         exit 0
     fi
     /usr/bin/logger -t logdiskcheck "WARN logdisk nearly full. Attempt to cleanup."
     # Force a log rotate.
     /usr/sbin/logrotate -f /etc/logrotate.conf
     # Delete more than a week old hprof files from component log directory.
     find "$log_dir" -mindepth 2 -mtime +7 -type f -name '*.hprof' -delete

root@vc05 [ /etc/cron.d ]# sh -x /usr/sbin/logdiskcheck.sh
      + log_dir=/var/log/vmware/
      + max_usage_percent=90
      ++ df /var/log/vmware/
      ++ awk '{print $5+0}'
      ++ tail -1
      + usage_percent=100
      + '[' 100 -lt 90 ']'
      + /usr/bin/logger -t logdiskcheck 'WARN logdisk nearly full. Attempt to cleanup.'
      + /usr/sbin/logrotate -f /etc/logrotate.conf
      error: error creating output file /var/log/vmware/vpostgres/pg_archiver.log.stderr.1: File exists
      error: error creating output file /var/log/vmware/vpostgres/pg_archiver.log.stdout.1: File exists
      error: error creating output file /var/log/vmware/vpostgres/postgres_cron.log.1: File exists
      + find /var/log/vmware/ -mindepth 2 -mtime +7 -type f -name '*.hprof' -delete
root@vc05 [ /etc/cron.d ]# df /storage/log
      Filesystem             1K-blocks     Used Available Use% Mounted on
      /dev/mapper/log_vg-log  10181944 10164068         0 100% /storage/log

This is NORMALLY not needed, VMware manually kicks logrotation every 10min if filesystem /storage/log/vmware is filled more than 90% BUT BE AWARE: not all log files are maintained by vmware config, ours is not!!!

5.16.4 delete logfiles manually (last option)

  • big file is not used by any process (lsof big-file)
  • logrotate script (step above) did not cleanup the disk filling problem
root@vc05 [ /storage/log/vmware/cis-license ]# ls -lah vmware-cis-license-gc.log.1
     -rw-r--r-- 1 root root 9.7G May 21 12:39 vmware-cis-license-gc.log.1

root@vc05 [ /storage/log/vmware/cis-license ]# lsof vmware-cis-license-gc.log.1 #file must not be in use

# truncating log is better then deleting it. logrotate may expect proper file numbering
root@vc05 [ /storage/log/vmware/cis-license ]# > vmware-cis-license-gc.log.1

root@vc05 [ /storage/log/vmware/cis-license ]# ls -lah vmware-cis-license-gc.log.1
     -rw-r--r-- 1 root root 0 May 21 12:53 vmware-cis-license-gc.log.1

root@vc05 [ /storage/log/vmware/cis-license ]# df -hT .
     Filesystem             Type  Size  Used Avail Use% Mounted on
     /dev/mapper/log_vg-log ext4  9.8G   58M  9.2G   1% /storage/log

5.17 Access/Troubleshoot PostgreSQL DB

  • fetch credentials as root user
cat /etc/vmware-vpx/embedded_db.cfg
  • swith into disabled vpostgres user
su -s /bin/bash vpostgres
  • connect database
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

5.17.1 log purge on vCSA 6.x

https://kb.vmware.com/s/article/2110031
curl -s "https://kb.vmware.com/servlet/fileField?entityId=ka234000000TvzOAAS&field=Attachment_3__Body__s" > 2110031_Postgres_task_event_stat.sql

curl -s https://raw.githubusercontent.com/lamw/vghetto-scripts/master/shell/queryVCDBvPostgres.sh | sh

/opt/vmware/vpostgres/current/bin/psql -U postgres -v TaskMaxAgeInDays=7 -v EventMaxAgeInDays=7 -v StatMaxAgeInDays=7 -d VCDB -t -q -f 2110031_Postgres_task_event_stat.sql

curl -s https://raw.githubusercontent.com/lamw/vghetto-scripts/master/shell/queryVCDBvPostgres.sh | sh

5.17.2 purge vcsa logs by truncating (not verified by VMware)

  • If db is full then regular kb will lead into error because temporary space is needed but not available
  • Find an idea what to do and how others do get around the problem:
https://communities.vmware.com/thread/560901
https://kb.vmware.com/s/article/1025914
  • get an idea on what to delete
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c '\dt
  • prepare truncate psql statement as you need copy paste into sql prompt:
[chris@lts1 ~]$ seq 100 | while read nr; do echo "truncate table vpx_event_$nr cascade;"; done
truncate table vpx_event_1 cascade;
[...]
truncate table vpx_event_100 cascade;
  • Open SQL Prompt and paste the comand from step above


  • Example db login:
http://www.postgresqltutorial.com/postgresql-cheat-sheet/
 /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
    psql.bin (9.4.9 (VMware Postgres 9.4.9.0-4442725 release))
    Type "help" for help.
    VCDB=#
        SELECT * FROM VPX_VERSION;
        SELECT * FROM VPX_HOST;
        \q

5.18 vCSA troubleshooting KBs

5.19 vCSA Important KBs

5.20 VCSA KBs to review

5.21 vCSA KBs to discuss

6 More fun

6.1 ESXi console

6.2 OpenSSL comandline

6.3 LAB exercises to discuss

mount | column -t

ls -lR / >std_out.log 2>err.log
cat std_out.log 
cat err.log

lsof -i -P -n | grep LISTEN | egrep -v '::1|127.0.0.1' | awk '{print $1 " "$9}' | sort -u

find /var/log /storage/log -follow  -type f -mmin -5

find /var/log /storage/log -follow  -type f -mmin -5 | xargs file | grep ASCII | cut -d: -f1 | xargs tail -f
lsof | egrep 'log$|out$' | awk '{print $10}' | sort -u | xargs tail -f | egrep --color=always -i '$|error|crit|warn'

lsof /var/log /storage/log | awk '{print $9}' | sort -u  | xargs tail -f | tee -a /tmp/all.log

reset

curl ifconfig.me

lsof /storage/log/vmware/vapi/endpoint/*



# log your work
some-important-cmd | tee -a /root/debug.log
script -a /root/my_session.script
Retrieved from "http://www.bitbull.ch/wiki/index.php?title=VCSA_6x_Troubleshooting_Free&oldid=5655"