New! Jupyter Lab Alpha Preview

Now that we’re back in business, we added a jupyter lab instance which is an alpha preview of the new interface being developed for jupyter notebooks. You can go to https://jupyterlab.crl.coloradomesa.edu and sign in with your CS Server Room domain account to provision your own notebook to play with.

All files saved within jupyterlab notebooks are saved to the same storage backend as your normal jupyter notebooks at https://jupyter.crl.coloradomesa.edu.

You can even run both notebooks at the same time!

For those who are note familiar with Jupyter Notebook, you can read up on it here: http://jupyter.org/

Here’s what the new jupyter lab interface looks like:

Poll: Storage Requirements For VMware VMs

https://www.facebook.com/groups/137436689669934/permalink/1501374413276148/

How much storage space should the average student need to start out with when using virtual machines? On average I see about 2-3VM’s pop up on a user’s account, nothing more than 6-8 VMs period.

Windows machines usually take up the most space, with the minimum being 32GB and up to 128GB for the main disk. Linux VM’s can use as little as 16GB up to roughly 64GB.

I’m thinking of setting up a per-user storage provisioning system, and maybe 200GB is a good start? Thoughts

Problems and Troubleshooting – Part 1

I decided I should start writing about some of my findings in the CRL or in my homelab when it is relevant to what we do here. It would be beneficial to those looking to learn more about working with server software and hardware.

This is NOT meant to scare anyone off, even if sometimes these turn into miniature rants. I’ll try my best to explain my thought process and troubleshooting.

Some of these things I will try to cross-post on my own website at https://jtcressy.net

Database Conundrums

Somehow, the vcenter server magically stopped starting the vpxd service. Rather, I found this out after suddenly getting 503 errors for the vpxd webserver pipe when trying to access the main web client. Starting it manually just resulted in it crashing repeatedly. (You can SSH into the vcenter appliance with the root account and run ‘service-control –start vmware-vpxd’ or get it’s status by running ‘service-control –status vmware-vpxd’).
What I needed to do was go look at the logs. After a bit of googling, I found out that they were located in /var/log/vmware/vpxd/vpxd.log. I would recommend looking at the log with either ‘less’ or ‘vim’ so that you can scroll up aswell as down. You can also look for the word ‘error’ by typing ‘?error’ in vim or less. It should just be at the bottom of the log since it dumps a crash report and dies. I found some key words in the error such as “pk_vpx_vm_virtual_device” right after the words “duplicate key value violates unique constraint”.

Googling this for a little while gave me this:
https://communities.vmware.com/thread/547479

  • This is a bug in 6.5 that is fixed if you are updated to 6.5b (wasnt quite up to date yet, lol)
  • Vmware is trying to add its inventory to the database and is failing because an entry is already there
  • Usually a result of changing a vm while vcenter is down or offline. It could even be as small as powering on or off a vm.
  • Workaround is to remove the offending VM from the inventory of the host it’s residing on and re add it later once vcenter is working

One small problem: I have no idea what VM it is.

Now, i looked up a little further to find out what data it was trying to write to the table. I saw somewhat useless information like what type of device it was and other variables for the device, but I wanted to find a field that linked it back to the main virtual machine, that turned out to be the ID field.

So, ok, we have a VM ID, but how do we find out what vm it is? the hosts don’t keep a record of what ID a vm has in vcenter. This ID also pertains to what vmware calls a “Managed Object Reference”. For example, vmrc url’s use “?moid=vm-xxx” to specify what vm to connect to over vmrc. Thus, i need to query the database for the VM id in that field.

While in the terminal of the appliance, you can invoke the postgres tool to manipulate the database. it allows you to connect without a password as long as you use the “postgres” username. The ‘psql’ binary is not in the PATH, so we have to invoke it directly with ‘/opt/vmware/vpostgres/current/bin/psql’ and to connect and use the postgres user, run ‘/opt/vmware/vpostgres/current/bin/psql -U postgres’.
Now that i’m in the interactive prompt, i can check that the VCDB exists with ‘\l’ then switch to that database by typing ‘\connect VCDB’.
Query the table ‘vpx_vm’ (list tables with ‘\dt’ and check my spelling) and filter for the VM’s ID by running “SELECT * FROM vpx_vm where id = ‘<vmid>’;” and replace <vmid> with the ID you found in the vpxd log. In my case the ID was 784 and I was able to scroll through the output far enough to catch a glimpse of the VM name. I then checked all of my hosts for that specific VM and powered off then unregistered the vm. If you are doing this in a huge datacenter with hundreds of hosts, I feel sorry for you. Hopefully you’d have a sane HA setup that allows you to afford the time to fix it. You’d be better off calling vmware support in a deployment like that.

 

DNS Failover – NOT!

vCenter appears to ignore the secondary dns server in the configuration. This means if your first DNS server goes down, vcenter will flip out. Hosts will disconnect, services will fail to start, etc. A workaround besides fixing that broken DNS server is to go to the configuration at https://<yourvcenter>:5480 and set the primary DNS to your other working DNS server and FULLY REBOOT vCenter.
Another way to remedy this would to be to run a number of servers in a failover IP configuration where a single IP address points to all servers in a cluster.
Shoutout to vmware for making us use complicated workaround setups.

Virtual Machine Consoles Now More Accessible

If you run the standalone VMRC client or VMware workstation, we have good news!

You can now connect to your virtual machine’s console directly to enable keyboard capture for advanced shortcuts. You no longer have to use the web console except for convenience. This will work on lab computers with VMware Workstation installed, aswell as personal laptops on the CMU WiFi.

We cannot guarantee if it will work off-campus.

To learn more about connecting to the CRL vcenter using VMware Workstation, Click Here

For VMware Fusion on macOS/OSX, Click Here

VM Templates

Templates for popular operating systems are being created on the vSphere server so that students can quickly deploy a VM to use in their project.

More information here: https://github.com/coloradomesa/cs-labs-public/wiki/VM-Templates

How to Get Started

Getting Started
For those of you needing to request accounts and get logged in to vmware, you can submit an account request using the link to the form at the top of this page.
After you receive an email from one of the lab assistants, use the “Change Your Domain Password” link at the top to change the temporary password we gave you.
You’ll have to change your password before you can log in.

For more information, visit our FAQ on github. We’ll be adding more information to this as time progresses.
For any questions, feel free to contact a lab assistant.

We will be using the cs-labs-public repo to keep track of issues regarding the server room, click the link titled “Open an Issue on GitHub” on the help menu at the top of this page to visit the issues section.
If you have any issues with your virtual machines or containers or have trouble accessing the vSphere web client or OpenShift Console, open an issue here and we’ll be able to sort it out.

Adding a ticketing system (WIP)

A ticketing system has been added for issues regarding the server room like user accounts, forgotten passwords, problems with the vsphere client, and more. It’s a work in progress, since automated emails have not been setup yet. So if you create a ticket, you wont yet get an email about creating it and the lab assistants wont get emails either.

If you need to request an account, please use the “Account Request Form” link at the top of this page.

I will update either this post or create a new post once it’s fully operational.

The ticket system is open source, and is based on osTicket (osticket.com)

Posted in: Uncategorized

Networks Upgraded, Functionality Restored, Password Change Implemented

Hi everyone!

Over the break we committed some network upgrades, but also had some issues and had to reset the network configurations on the vmware servers. The configuration is the same, but we had a roughly 2-week outtage and all services were non-functional. Apologies if your VM was abruptly shut down, hopefully there will be no problems after powering back on.

As of now it’s all fixed and back to the way it’s supposed to be working, with upgrades under the hood. We have a new switch, more ethernet connections, optimized VLAN topology, and added a way for you to change your domain/vmware password.

A link to a web-based password change application is at the top of this page on the nav bar titled “Change Domain Password”. For any new accounts created, we will be supplying temporary passwords going forward and you will need to change your password using the aforementioned link before you will be able to login to VMware or any other services that use a domain login (including local machines in the lab).

We moved the netgear switches to Beta rack and Delta rack, so now all three racks have networking. The new alpha rack switch is an HP Procurve 2800 series and is much better to configure than the Netgear switches. It even has more ports!

Upon request, we can configure certain ports or groups of ports to specific VLANs (virtual local area network) for projects that can be hosted in the server room. If you have a physical machine that you want to segregate away from the rest of the network, we can make it happen. Two of the most popular networks in the server room are ACM and CSC, accessible by separate wifi SSID. These are two separate vlans that can also be configured per-port on each switch upon request. Contact a lab assistant or CS professor for more information.

Posted in: Uncategorized

Updates – vSphere Client Fixed, SSL Support Added to Entire CRL Domain

The vSphere Web Client is now working 100% again, accessible at its normal URL.

In addition to the fixes, I have also introduced HTTPS encrypted with Let’s Encrypt certificates for all current and future web services. If you want to host a web app on our network, we have a reverse-proxying load balancer in our docker environment that can proxy any host in the server room with HTTP *AND* HTTPS. A lets encrypt app sits in a container and runs domain verifications, so yoursubdomain.crl.coloradomesa.edu can be verified and an SSL certificate issued. The SSL certificates sit on the persistent storage behind the containers, and can be retrieved to be used externally if necessary. Each certificate will be on its own, one private key per domain. Most of the time, SSL certs will be used by the load balancer as it has to decrypt HTTP requests to read the host header before determining whose server to forward the request to.