@bwhite1234 wrote:
Host crashed due to power failure and bad UPS, now VIRL VM powers up, can log into Ubuntu VM, but VIRL services are not working. Basic web server works such as downloading Maestro clients, but can't log into UWM (page can not be displayed).
How do you troubleshoot VIRL services, or just delete VM and redeploy from OVF again?
virl@virl:~$ sudo virl_health_status
Disk usage:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/virl--vg-root 71G 39G 32G 56% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 16G 4.0K 16G 1% /dev
tmpfs 3.2G 3.8M 3.2G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 16G 64K 16G 1% /run/shm
none 100M 4.0K 100M 1% /run/user
/dev/sda1 236M 79M 146M 36% /boot
ramdisk 16G 0 16G 0% /var/lib/nova/instancesCPU info:
8 Intel(R) Xeon(R) CPU E5620 @ 2.40GHz cores
Load: 5.5%, 4.4%, 2.6% for the past 1, 5, and 15 minutes
Overcommitted to 32 cores (multiplier 4.0)RAM info:
ERROR 2016-04-15 19:32:45,473 virl.health Failed to pack status of a health check
Traceback (most recent call last):
File "/home/jenkins/jenkins/jobs/VIRL_CORE_build/workspace/test-virl-repo/virl/health/__init__.py", line 44, in get
File "/usr/local/lib/python2.7/dist-packages/werkzeug/utils.py", line 73, in get
value = self.func(obj)
File "/home/jenkins/jenkins/jobs/VIRL_CORE_build/workspace/test-virl-repo/virl/health/resources.py", line 91, in status
File "/usr/local/lib/python2.7/dist-packages/werkzeug/utils.py", line 73, in get
value = self.func(obj)
File "/home/jenkins/jenkins/jobs/VIRL_CORE_build/workspace/test-virl-repo/virl/health/resources.py", line 71, in data_mf
File "/home/jenkins/jenkins/jobs/VIRL_CORE_build/workspace/test-virl-repo/virl/common/hosts.py", line 1085, in get_ram_required
File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 516, in loads
return defaultdecoder.decode(s)
File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Expecting value: line 1 column 1 (char 0)NTP servers:
pool.ntp.org iburst
us.pool.ntp.org iburstremote refid st t when poll reach delay offset jitter
*198.55.111.5 204.123.2.5 2 u 51 64 37 42.701 -2.343 1.118
204.2.134.162 44.24.199.34 3 u 49 64 15 59.997 5.230 79.739
128.138.141.172 .ACTS. 1 u 46 64 37 11.168 -3.674 1.072Interface addresses:
1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
2: eth0 inet 10.0.1.231/24 brd 10.0.1.255 scope global eth0\ valid_lft forever preferred_lft forever
3: eth1 inet 172.16.1.254/24 brd 172.16.1.255 scope global eth1\ valid_lft forever preferred_lft forever
4: eth2 inet 172.16.2.254/24 brd 172.16.2.255 scope global eth2\ valid_lft forever preferred_lft forever
5: eth3 inet 172.16.3.254/24 brd 172.16.3.255 scope global eth3\ valid_lft forever preferred_lft forever
6: eth4 inet 172.16.10.250/24 brd 172.16.10.255 scope global eth4\ valid_lft forever preferred_lft forever
44: virbr0 inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0\ valid_lft forever preferred_lft forever2: eth0: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 00:0c:29:27:fa:f3 brd ff:ff:ff:ff:ff:ff
3: eth1: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 00:0c:29:27:fa:fd brd ff:ff:ff:ff:ff:ff
4: eth2: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 00:0c:29:27:fa:07 brd ff:ff:ff:ff:ff:ff
5: eth3: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 00:0c:29:27:fa:11 brd ff:ff:ff:ff:ff:ff
6: eth4: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 00:0c:29:27:fa:1b brd ff:ff:ff:ff:ff:ffHypervisor info:
QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3~cloud0), Copyright (c) 2003-2008 Fabrice Bellard
INFO: /dev/kvm exists
KVM acceleration can be usedBridge patch applied:
Bridge module patch:
For kernel 3.19.0-25-generic: NOT applied!
For kernel 3.19.0-33-generic (running kernel): AppliedMySQL is not available
Salt ID: B2659C6D.virl30.info
Salt Master: ['salt-master.cisco.com', 'salt-master-2.cisco.com']
Salt Ping: SuccessHost salt-master.cisco.com:
DNS: Success - 128.107.0.207
Ping: Success
Port 4505: Success
Port 4506: Success
Connect: SuccessRabbitMQ status:
DEBUG 2016-04-15 19:32:52,416 amqp Start from server, version: 0.9, properties: {u'platform': u'Erlang/OTP', u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'capabilities': {u'basic.nack': True, u'publisher_confirms': True, u'consumer_cancel_notify': True, u'per_consumer_qos': True, u'consumer_priorities': True, u'exchange_exchange_bindings': True, u'authentication_failure_close': True, u'connection.blocked': True}, u'copyright': u'Copyright (C) 2007-2014 GoPivotal, Inc.', u'version': u'3.4.3', u'cluster_name': u'rabbit@virl'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']
DEBUG 2016-04-15 19:32:52,418 amqp Open OK!
DEBUG 2016-04-15 19:32:52,431 amqp Start from server, version: 0.9, properties: {u'platform': u'Erlang/OTP', u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'capabilities': {u'basic.nack': True, u'publisher_confirms': True, u'consumer_cancel_notify': True, u'per_consumer_qos': True, u'consumer_priorities': True, u'exchange_exchange_bindings': True, u'authentication_failure_close': True, u'connection.blocked': True}, u'copyright': u'Copyright (C) 2007-2014 GoPivotal, Inc.', u'version': u'3.4.3', u'cluster_name': u'rabbit@virl'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']
DEBUG 2016-04-15 19:32:52,432 amqp Open OK!
[{pid,5500},
{running_applications,[{rabbit,"RabbitMQ","3.4.3"},
{os_mon,"CPO CXC 138 46","2.2.14"},
{mnesia,"MNESIA CXC 138 12","4.11"},
{xmerl,"XML parser","1.3.5"},
{sasl,"SASL CXC 138 11","2.3.4"},
{stdlib,"ERTS CXC 138 10","1.19.4"},
{kernel,"ERTS CXC 138 10","2.16.4"}]},
{os,{unix,linux}},
{erlang_version,"Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:8:8] [async-threads:30] [kernel-poll:true]\n"},
{memory,[{total,74184496},
{connection_readers,192696},
{connection_writers,32256},
{connection_channels,81872},
{connection_other,276832},
{queue_procs,266104},
{queue_slave_procs,0},
{plugins,0},
{other_proc,13775432},
{mnesia,100072},
{mgmt_db,0},
{msg_index,52944},
{other_ets,775624},
{binary,36797408},
{code,16403685},
{atom,561761},
{other_system,4867810}]},
{alarms,[]},
{listeners,[{clustering,25672,"::"},{amqp,5672,"0.0.0.0"}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,13438722048},
{disk_free_limit,50000000},
{disk_free,33724370944},
{file_descriptors,[{total_limit,924},
{total_used,17},
{sockets_limit,829},
{sockets_used,15}]},
{processes,[{limit,1048576},{used,287}]},
{run_queue,0},
{uptime,400}]RabbitMQ configured for Glance and Neutron is available
RabbitMQ configured for Nova is availableOpenStack system services:
[ ? ] apport
[ ? ] console-setup
[ ? ] cryptdisks
[ ? ] cryptdisks-early
[ ? ] dns-clean
[ ? ] irqbalance
[ ? ] keepalived
[ ? ] kexec
[ ? ] kexec-load
[ ? ] killprocs
[ ? ] kmod
[ ? ] lightdm
[ ? ] mysql
[ ? ] networking
[ ? ] ondemand
[ ? ] pppd-dns
[ ? ] qemu-kvm
[ ? ] rc.local
[ ? ] sendsigs
[ ? ] umountfs
[ ? ] umountiscsi.sh
[ ? ] umountnfs.sh
[ ? ] umountroot
[ ? ] zfs-fuse
virl (controller): all required services are running.OpenStack compute service for STD is not available
OpenStack identity service for STD is not available
OpenStack network service for STD is not available
OpenStack image service for STD is not availableOpenStack compute services:
Failed to get OpenStack compute services list.OpenVPN services:
OpenVPN server:
Disabled
Stopped
Status:
No status information presentAutoNetkit services:
virl-vis-processor running
virl-vis-webserver listening on port 19402
ank-cisco-webserver listening on port 19401
virl-vis-mux runningVIRL environment priority (lowest->highest): global conf, local conf, SHELL env, CLI args
Global config can be defined at "/etc/virl/virl.cfg"
Local config can be defined at "/home/virl/virl.cfg"
To set as SHELL ENV var: export NAME=value
To unset as SHELL ENV var: unset NAME
=========================================================
Your global config:
VIRL_STD_DIR = /var/local/virl
VIRL_STD_HOST = 0.0.0.0
VIRL_STD_PROCESS_COUNT = 20
VIRL_STD_PORT = 19399
VIRL_STD_USER_NAME = uwmadmin
VIRL_DEBUG = False
=========================================================
Your local config:
=========================================================
Your SHELL environment:
=========================================================
Used values:
VIRL_STD_DIR = /var/local/virl
VIRL_STD_PORT = 19399
VIRL_STD_USER_NAME = uwmadmin
VIRL_STD_PROCESS_COUNT = 20
VIRL_STD_HOST = 0.0.0.0
VIRL_DEBUG = False
=========================================================
Exception TypeError: "'NoneType' object is not callable" in removeHandlerRefpatch at 0x7f321d1b09d0> ignored
STD/UWM is initialized with the following users: uwmadmin,guest
STD server on url http://localhost:19399 seems to be broken (returned status code 503)
UWM server on url http://localhost:19400 is down
Webmux server on url http://localhost:19403 is listeningSTD server version:
STD server licensing:
STD server autonetkit status:
virl@virl:~$
Posts: 1
Participants: 1
