Posted on

A short discussion on VMware’s Army of Service Providers and MSP’s

I’ve had an off and on relationship with the VMware Service Provider offerings over the last few years. I’ve worked for a partner, then on to VMware Perpetual Sales and now with VMware Service Provider division. I came back because I absolutely love the offering.

When I worked for a Service Provider Partner, this program was called the VMware Service Provider Program. Simple enough. Then the name was changed to vCloud Air Network. This was roughly around the time of vCloud Air becoming available. I’m not going to lie, it was pretty confusing for customers. With that confusion and some opinions on vCloud Air as a product, I’m curious if customer automatically assumed vCAN had the same reputation.

Now, vCloud Air has been sold to OVH. Who, funny enough is a VERY large vCloud Air Network Partner. It was a huge win-win in my book and really gave OVH more leverage in the US market. You see, OVH is killing it in Europe. So, when presented with the opportunity to onboard large set of new customers and operationally take on new data centers in one shot, it provided a unique opportunity.

So, this week, VMware announced that the vCloud Air Network program will now be called the VMware Cloud Provider Program. A smart move in my book to distance from the vCloud Air name as well as really driving their purpose in this Hybrid Cloud future VMware is painting. VMware wants to empower these partners and our customers with the tools to provide a great experience, whether its Infrastructure as a Service or Managed Services for those partners that want to go above and beyond. This Service Provider Program is over 4500 strong and growing each quarter.

Now, with last years announcement of VMware Cloud on AWS and it becoming generally available this week you wouldn’t be wrong to think this is another situation where VMware is competing against the service provider community. BUT! I challenge you to take another look with me.

VMware Cloud on AWS is a great offering with the ability to stand up new environments on the fly. It will help companies take advantage of AWS services using an infrastructure model they are intimitaly familier with. The idea as I see it is that companies with an IT workforce, have the engineering know-how to utilize such a service, but companies with minimal IT or who are already running on a service providers offering via MSP model may not know where to even begin.

This is where the MSP model in the future will step in. VMware is working to empower our MSP partners to resell the VMConAWS service, so they can continue to support their customers but offer the same flexibility the larger customers will now begin to enjoy. It’s not a competition when we still want our MSP partners to grow in this model with us.

I’m looking forward to watching these offerings grow, especially in how we offer them to our partners. The VCPP and MSP future is bright.

Posted on

Technical Compliance and vCloud Air Network

For the last year and a half, I’ve been working as a Systems Engineer for VMware’s Commercial segment. I’ve really enjoyed the time on this team, engaging customers in the field and getting comfortable with VMware’s sales practices. During this time, my boss had recommended me for a potential vCloud Air Network position because of my background working for a vCAN partner. That recommendation didn’t pan out but it did trigger a serious consideration for vCAN as my next move.

As of this month, I am official the Technical Compliance Systems Engineer. Odd title but what it means is that I will be focusing on Usage Meter and helping build a better reporting process for VMware’s Service Provider network.

I will be focusing on Usage Meter and program compliance. Its a pretty neat role in that this is the first one in the company. It will be interesting in that I have a chance to define the role as the program grows. As the vCAN moves forward, I believe there is definite need to make changes to the reporting process and the data that is collected. I have a few ideas already and look forward to diving in. Some of the things I am excited about is creating content for the SP’s and assisting on changes to the program that benefit everyone involved. From what I’ve seen so far, SP’s should be very excited.

Posted on

Homelab: Compute

A little while ago, I wrote up some goals for the Homelab. The idea behind these goals was not so much to build a lab that compares in power to a normal DC, but to build something that can match in the way its configured. This is for my own continued education as well as for use in Demo’s of products to customers.

So lets take a look at the one of the areas mentioned in more specific terms. Compute. In my Goals post, I listed the following items as goals to look for. I must admit, I already had a product family in mind by the time I made these goals, just not a specific model chosen. Never the less, lets revisit that list.

Compute:

  • 1U rack height
  • 4 or more Cores
  • Dual NIC’s
  • 1-2 PCIe slots
  • Greater than 32GB RAM Max
  • IPMI (Dedicated or Shared NIC)

 

What Matters Most?:

In the case of my lab, it matters more to be able to match a configuration than it does to have the most powerful one ever. As an Engineer at heart and a Pre-Sales Systems Engineer, its important to work through configurations for accuracy to how customers deploy their infrastructure. More specifically, how customers deploy ESXi. Everything from the vmkernel IP’s to advanced settings on processor performance.No as most people would consider, I could have done this in a nested environment. Nested is a great idea when you are studying for test or wanting to do functional testing, but long term it has its limits. Nested was too underpowered and giant dual socket, 6-8 core servers were too power hungry! Also as a side note, this was going to be in my office and noise also needed to be taken into consideration quite a bit.

 

Core Count:

I decided that a 4-core machine with a single socket was plenty powerful. Take a look at all of the Intel NUC blog posts and you’ll see that they have what it takes in power, but are a little light on RAM. Spoiler alert: the only time i’ve hit almost 100% CPU was during the vRA deployment. Even now, I have one box at 40% and two at 20%. 4-Cores are more than enough, even without right sizing the VM’s down.

 

Network Connectivity:

As most labs go, 1G Networking is sufficient. Its hard to saturate a 1G switch in a homelab by most peoples standards. When I started looking at hardware, I looked at the prices of 10G networking and it put me off. At the time, 10G network switches would have put me back about $1500, and those only have 8-12 ports. Though the cost has come down since then, it was insane. My plan was to buy hosts with PCI slots available for 10G cards later. My major concern here was that I needed to make sure that any server purchased, had two physical NICs onboard. I’ll hand it to William Lam for building that into the NUC’s, but this needed to be clean, built in and ready to go with minimal firmware work.

Now at the same time I was looking, Supermicro had just announced their X10SDV line of motherboards. These are Embedded Xeon boards with 10G Ethernet or SFP+ built in. Let me repeat that really quick… 10G Networking built into the motherboard! That was potentially going to save me about $200 down the line, or in my case cost me about $200 more now vs later if I bought them. This lends towards my goal of upgradability. I could start with 1G network switches now and only need to swap the switches and cables later to upgrade. tempting. It wasn’t necessary but would definitely get me closer to “real world server config” as possible.

 

Memory:

Downside to the Intel NUC’s is the 32GB or RAM. RAM will always be the bottleneck in the DC and its no different in the homelab. I needed to make sure that any servers I run could handle more than 32GB. Its a common complaint amounts the vExperts and its one I wanted to avoid. When I started, I really liked the Shuttle PC’s. Their form factor has been well known for years and they have made some advances in max RAM capacity. Also to note is that most of those with higher limits also had dual NIC’s, so that was a plus. When the Supermicro boards came out though, they blew the competition out of the water. 128GB of RAM max capacity in 4 DIMM slots. Downside there is cost. 32GB DIMMs at the time cost about $250 per DIMM, ouch! Altogether still needed to make sure I wasn’t limited.

 

OOB Management:

In a previous role, I used both iDRAC and IPMI. I was leaning towards IPMI, only because to get something with iDRAC, I would have had to sacrifice  for noise levels and power consumption. That wasn’t going to happen and there are plenty of boards out with IPMI now. An added bonus to going with IPMI was all of the open source central management solutions out there. In the past I’ve used xCAT, developed by IBM/Lenovo engineers and made available as open source to their server users. It gave me a CLI for managing a whole datacenter’s worth of hardware and uploading firmware to hosts from a single point of management. Ideally, I would want to do the same here. If IPMI is shared with a network port or the board has a dedicated port, that didn’t matter. What matters was getting ISO’s to hosts via the network and not using any KVM equipment.

 

End Result?:

When I started looking into what I would use for compute, the X10SDV’s were still just a marketing promise. While working on a couple ideas for whitebox configs, Supermicro went from marketing to production. For a short time, I considered doing an open-air deployment and doing something like the Ikea Helmer Render Farm or something closer to how some larger DC’s place motherboard on rack trays (this was something that seemed particularly interesting to me).

In the end I went with the SYS-5018D-FN8T (or X10SDV-TP8F Motherboard). I went with this specifically for:

  • 35W TDP
    • This plays into low power and how quiet the server runs.
  • 6x Dedicated 1G NIC’s
    • If you were going to start with multiple 1G NIC’s to put off 10G networking as long as possible
  • 2x 10G SFP+ ports
    • BOOM! Favorite reason #1
  • 128GB RAM Max
    • Favorite reason #2
  • Small form factor
    • Comes in the same 10″ deep server chassis that I was already looking at for whitebox configs.
  • PCI Expandability
    • This is more for future use, looking at a Supermicro HCL storage controller.
  • Dedicated IPMI
    • also… Supermicro has Central Management tools in abundance. Depending on how you want to access or what you want to manage.

 

The server is a little more expensive than I was initially interested in, but meets my power and noise requirements. Noise and power were huge concerns for me, when originally looking at used servers on Ebay. I had to also consider adding a new line to the office, given that the main circuit breaker was full from when the house was built, it would be run from the main box outside. So the SYS-5018D-FN8T helps conserve power which really worked for me, but still provided upgradability that I knew I would need. I had started with two servers initially, and 64GB of RAM each (2x 32GB DIMMs, with an internal discount for Kingston). Then held out a little longer for a 3rd node for HA and potentially vSAN later on. All in all I love these servers. They are doing a great job and holding up really well.

When building the homelab, just remember to think about what you truly want out of it. I chose low power, cooling management over just functional testing, especially since I use this to show customers. Depending on your personal “business value” that could be different.

 

As a side note:

I attempted to purchase all of the same parts for 5018D-FN8T, and its not worth the time and effort to assemble over the small saving. Buy the completed system and only add the active cooler fan if you think you will put a little extra stress on the machines.

Posted on

Homelab: Goals

I’ve been meaning to do a write up on designing my homelab. In my last job, I had access to hardware and some essential networking bits, but now that I’m a Field SE, i’m in a different situation. I have access to internal tools and nested deployments (otherwise called PODs), as well as some Hands-On-Labs deployments. These are great for doing quick demo’s but for continued education purposes, the consensus among SE’s is that “Nothing beats a homelab”.

Continue reading “Homelab: Goals”

Posted on

Professional Growth: Part 4 – Future Proofing

Intro text:

In keeping with the continued theme of this series, I wanted to know what the responders felt was a good technology path of choice, based on how the market is changing now and what areas they thought had the most potential. I know that the title of this post makes it seem like there is a specific technology horse to bet on at the races, but that isn’t the case. IT is broad and ever changing. These are the number of ideas people think, based on different perspectives within the market.

 

So the next question asked was “What areas of technology do you see the most growth potential that people aren’t considering?” The idea behind this is where can we fill the void, and what emerging technologies are people not hearing about just yet that could become the next buzzword of the tech conference world? Because you know we are all so very ready for the next IoT, Cloud, Hyper-Converged buzzword.

 

Considering I was looking to for a technology, I was surprised that half of the responses still centered on soft-skills. I think we’re seeing a trend with this, no? Granted there are some technology buzzwords that popped up like DevOps and Automation, that was only half of what came up.

So lets get the obvious stuff out of the way, shall we? DevOps. It has many meanings, but I think I like AgileAdmin’s best:

DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.

The idea that greater collaboration between the two teams can make applications more efficient, set expectations a bit closer to reality and create a more agile group of people who support the business. The key thing being speed. Faster, without the compromise of quality. Now given those two objectives, that likely means its more expensive! You know, the IT Holy Triangle “Fast, Good, Cheap, Pick Two”. Now in the case of today’s technologies, there isn’t a company or IT shop that hasn’t looked at or considered things like Chef, Puppet, Ansible or things like them. These are systems to help take applications from development to production as fast as possible, and are helping shape how we can make our applications more scalable. Which is HUGE!

Automation, the next on the list. Although most people think it falls into DevOps, thats not always the case. It depends on the situation and the business. The fact still remains though, if you work in technology and you allow yourself to do a task more than twice without trying to automate it, you aren’t doing the job justice. Sure scripting is hard, but professional growth isn’t meant to come easily and nothing easy is ever really worth it.

That kicks nicely into the next one, Scripting and Programming. This is actually becoming a focus for many institutions at the elementary level. Which is great to see so much focus on helping our younger ones become effective analytical thinkers! Programming and Scripting gives people a new perspective on problem solving. It helps people realize how to break down processes and find the issue. Even if you don’t work on something technical, learning to code can help you break down a workflow to find inefficiencies and that can be huge! Beyond that, we need to focus on building applications that scale and deploy in a more “Cloud-like” manner. Thus promoting business resiliency! That is the new focus in programming/scripting, because no one likes snowflakes anymore.

Software Defined Anything. From the infrastructure standpoint, this is less about trying to keep up with trends, and more to do with understanding new ways to support demand. One publication had an interesting bit for Networking guys. They posited that Network Engineers not trying to understand SDN technologies would mark them as Dinosaurs waiting for extinction. Not to say that I fully agree, but Architects in the Enterprise space should at least try to understand to see if it makes sense for their business, else they haven’t done a good enough job vetting out technologies (Which is the basis for having an Architect).

Security, big surprise right? This is actually a major issue given the high number of breaches that have been going on. Beyond that, you have to support the businesses compliance efforts. Ever sat in a audit meeting? They are boring and long. The idea being that someone is looking for potential vulnerabilities in your configurations and deployments that don’t meet a standard. No one likes them, but they are important to your customers, so hop to it, get on that security bandwagon. Who knows, maybe you’ll get to go to the blackout conference. I hear its crazy!

So what about the soft-skills? Well, one of my favorite responses was “you need to get out of your comfort zone to grow”, which is so true and not just for the obvious reasons. Ever heard of the Full Stack Engineer? Its a person who can talk about everything from the Infrastructure to the Applications. Understanding how to best support the different technologies that use the infrastructure and tuning everything in business to work well together. Its a hard concept to make a reality, but the easiest thing to start with is getting out of your comfort zone, and learning the ancillary technologies to the one you specialize in. If you cover compute, check out application and networking. If storage, how to work with the applications and network guys to decrease I/O but maintain or increase performance.

Business Skills! This one hits a bit close because its something I talk about with customers often. IT is made up of technology that consistently costs a lot of money. The costs have also continued to grow. People working in IT because of this are constantly thought of as a cost center, instead of a business enabler. Remember when trying new technologies was about making the business more efficient? Yeah, thats mostly forgotten now. In most settings, there are the expected technologies and everything after that is gravy, even if its about making the job of the IT person more efficient or just easier to save them time. The more that you can align your initiatives with the business issues, there more you produce a value. Most companies want automation to lower IT head count, while the IT organization wants automation to increase efficiency. The key is showing how certain actions or systems can reclaim your time, so you spend less on keeping the lights on and more time thinking ahead to better the business.

Cross Department learning. Now this adds on to the business skills portion, because IT needs to understand what others are doing. If we don’t try and spend a little time getting to know the people we support and the things they do, how can we expect to find better solutions for them? We have to become a stakeholder in their objectives and goals. Again, this fits into working with other tiers in the IT stack, the better we understand the parts we support, the more efficient the IT systems and their deployments can get (Not to harp on that again, but its important). You have to work together, if not, you’ll only be seen as a road block, but remember its a two way collaboration so they (the other team) need to buy in as well.

To quote “The First 20 Million is Always the Hardest”, the technologies or soft-skills should support the following idea:

Simplify, Clarify, Economize

Learning about systems or technology that can simplify and automate, Clarify the intent of technology in support of and aligning with the business, Economize by concentrating on technology that helps save the company money, or makes them more efficient at engaging with and handling more business. I think that sums it up pretty well! Plus I got to use a quote from a movie I love.

Posted on

VMworld First Timers!

First off, I am still alive, just acclimating to my new role and the travel with VMware. To get back into the swing of things and as I get ready for some new things coming up, I thought this would be a good return to writing post.

So for those of you preparing to go to your first VMworld, CONGRATS! You are headed to an excellent conference with roughly 27,000 other people! Yes, let that sink in for half a second… twenty. seven. thousand. people. And that is if they didn’t get additional people to show up this year!

So, for you first timers, I know how you feel. I was you last year, and these are the tips that I got and learned myself that I want to share.

  • SHOES: Get a pair of shoes that are comfortable for walking… 10-15 thousands steps a day in. If you don’t have a step tracker, then you should get one. If you decide to buy a new pair of shoes, BREAK THEM IN FIRST! That was my mistake, I didn’t.
  • SESSIONS: You won’t make them all, it just doesn’t happen. You can try and plan the perfect schedule full of amazing sessions. You’ll want to be there for them all, and then, you’ll walk into the Solutions Exchange. You’ll walk out and realize you missed half the day. Don’t feel bad, it happens to everyone.
  • EVENTS: Lets call it what it is, parties. Call your partners, VAR’s and Vendors and see who has an event going on. This is a great chance to relax, meet people and just get off your aching feet for 30 minutes to an hour. (Because there are about 10 parties happening each night)
  • GOING ALONE?: I did this last year, and although you are surrounded by 27,000 people, there is no weirder way to feel alone. You don’t have to feel that way though. Have you ever met someone that you can talk shop with? Talk about technology and you just go on and on for an hour or two, before you realize time has passed? Yeah, you have 27,000 other people there just like that. This is my single biggest piece of advice. At times you will be walking with a large crowd in the same direction, looking for a table to sit at and see a single chair at a table of 5 or maybe sitting at an event. The best thing you can do at VMworld when you feel alone, or anti-social, just turn to your left or right and introduce yourself. You will be amazed at the people you meet. Heck, a couple times the people I met were VMworld presenters!
  • KEYNOTE: Although its great to attend the main keynote speech in the main hall, there are additional keynotes each day. Instead of going to the main hall, go to the community space. Sit with the bloggers, check out vBrownBag, talk with the VMware Engineers at the “Office of the CTO” Booth (VERY COOL FUTURE TECH!). Get out of the main room and the absolutely insane crowd. Oh and don’t expect to have great mobile data service during this time.
  • COMMUNITY: This is the real reason people go. At my last job, my coworker helped persuade my boss to send me to VMworld (THANK YOU!!) and this was his tip. Meeting with and networking in the community is so much more valuable than just attending the sessions (which are recorded and put on youtube). He was so very right. The people that I met at events, while walking towards a session and even in the community lounge I’ve continued to talk with via twitter and slack. Amazing people!

Thats it for now, but for those heading to their first VMworld, enjoy it! Its a great experience and I recommend hitting up all of the community based parties and events, those were my favorite.

Posted on

Ramping up at VMware

I’m on week 3 at VMware, working my way through training and tasks designed to get me “Ramped Up” in the role. Its been incredible so far and for so many more reasons than I expected. First, let me say that I’m truly sorry I haven’t done the next Professional Growth post, I’m actually going to combine question 4 into post 3, and do a repost so look out for that. I’ve been a little busy but I promise to get back on track with those.

First, lets get this question out of the way.

“You went to VMware amidst the Dell Acquisition? Why?”

Yes, this was a major concern even before getting a call to do the first interview. There are so many articles and blog posts centered around VMware and Dell’s acquisition of EMC. Yes, there was a recent round of layoffs at VMware that was pretty significant. Ultimately, I felt comfortable with the stance VMware is showing and reached out to multiple contacts both inside and outside of VMware to get opinions.

I walked away from those conversations thinking that VMware, though part of the Federation, is very strong in its own right. This company still has new areas to grow and I’d like to be a part of that.

Alright, now that that is behind us, moving on.

Going to the “Dark Side”

I’ve heard this numerous times from my new coworkers and a few others. I understand that becoming a Pre-Sales System Engineer means that I am convincing companies that Product X is right for them, but its so much more than that. At the end of the first week, I questioned why that phrase is even used at all. By the end of the second, lets just say I don’t see it. This is an excellent opportunity and allows me to see and help so many people in so many different environments. I’m here to help validate, demo, show the value of these products and help solve problems. I don’t see a dark side to it at all.

First Impressions on my team and role

I wasn’t exactly sure what to expect going in. This is a vendor role, remember. Up until now, I’ve been a customer. One of the first things that struck me was all of the members willingness to help. People in different areas of the business have gone out of their way to get me slide decks and 1-on-1 meetings to discuss products, helping me get up and running.

My account reps took a good chunk of time to discuss and exchange knowledge and ideas, during a very busy part of their quarter, when they should be focusing on closing deals I obviously am not a part of. Above that, just being available far more than I expect them to be.

But, the best part of this actually happened around going to a customer’s business to do a demo. Now, as I mentioned earlier, I’ve been told “welcome to the Dark Side” plenty of times. This is part of sales, I support sales by doing the technical portion. It’s an understanding that I will convince companies that they need some software to solve their problem or reach their goal. What I wasn’t completely expecting was hearing the account rep and current systems engineer push on the idea of “we don’t sell shelf-ware”. The idea is that we don’t push products you don’t need to solve your current problem, or isn’t part of a soon to be project/goal.

That in itself makes this move all the sweeter. Why? Becuase it goes back to a core competency at my last job, a type of company culture if you will. At Voxeo/Aspect, they called it customer obsession. Doing what is right for the customer and seeing it through to the end. I’m reminded of that by this team and its amazing to see that idea being pushed, even in Pre-Sales.

Drinking from the firehose

Yeah, I have a lot of products to catch up on, but I’ve never been one to NOT want to learn a new enterprise tech. BRING. IT. ON.

Work | Life Balance

Coming from a customer background, I was presently surprised when my phone died the other night and I didn’t have to worry about an on-call rotation. But what really surprises me, is when my manager ends a friday call with “Alright, have a good weekend and remember, Family first… Work Second”. Work/Life balance is incredibly important here and as I agreed to this role, my one concern was how it might affect mine. I had spent a considerable amount of time working on this in my last job to improve it, to the best of my ability. While at VMware, its practically pushed in my favor by management.

Summary:

All in all, I already considered having VMware on my resume as a great career booster, but the perks keep rolling in. I don’t mean that in a way that suggests good discounts or benefits, which coincidentally are also great. The culture is good, the team is great and the role  will definitely cater professional growth beyond what I initially considered. I look forward the future that is at VMware. Now if I could only come up with a paper for VMworld… seriously, what should I talk about?!?!

Posted on

From Customer to Vendor, I’m making the jump!

While I have been writing posts on professional growth, I’ve also been working on what my next step should be. Like anyone trying to continue their career growth, you take a few things into consideration and draw up a plan.

  • What is the larger end goal for my career?
  • What knowledge and experience am I missing to get there?
  • What short term goals and next steps can I do to help?

It’s with those ideas in mind that I am happy to say I will be joining VMware in March, as a Systems Engineer!

This is a role I felt would greatly challenge me and continue to push me forward into new markets and challenges. It also doesn’t hurt that it’s an excellent team at an already amazing company. I am extremely excited for this opportunity and what it entails for my future.

As I continue forward on this journey, I’ll continue to blog what I see and do. In the meantime… VMware here I come!

Posted on

This Method Requires Authentication – Full Version

We were having some issues with one of our VCSA’s and creating or subscribing to Content Libraries. So here is our resolution.

Symptoms:

  • When creating a local Content Library, when clicking finish, it errors with: This Method Requires Authentication
  • When subscribing to another Content Library that has authentication disabled, after copying the json URL into the field and clicking next, it halts the view and states: This Method Requires Authentication
  • When attempting to download Support Bundles from the VAMI at https://<vCenter FQDN>:5480, Downloads timeout and fail

SSH into the VCSA and check the following log files:

/storage/log/vmware/vdcs/cls.log
/storage/log/vmware/vdcs/ovf.log
/storage/log/vmware/vdcs/ts.log

In cls.log, you will be looking for something like this:

cls.log
=========
2016-01-20T14:18:01.773Z | DEBUG    | unset-opId       | tomcat-http--39           | SsoOverRestVerifierUtil        | Trying to verify request signature using following; host:<vCenter FQDN>, port: 443, uri:/cls/resourcebundle
2016-01-20T14:18:01.800Z | ERROR    | unset-opId       | tomcat-http--39           | SamlTokenImpl                  | Signature validation failed
javax.xml.crypto.dsig.XMLSignatureException: the keyselector did not find a validation key
        at org.jcp.xml.dsig.internal.dom.DOMXMLSignature$DOMSignatureValue.validate(Unknown Source)
        at org.jcp.xml.dsig.internal.dom.DOMXMLSignature.validate(Unknown Source)
        at com.vmware.identity.token.impl.SamlTokenImpl.validateSignature(SamlTokenImpl.java:653)
        at com.vmware.identity.token.impl.SamlTokenImpl.validate(SamlTokenImpl.java:535)
        at com.vmware.vim.sso.client.DefaultTokenFactory.parseToken(DefaultTokenFactory.java:46)
        at com.vmware.vim.sso.http.impl.AuthVerifierImpl.validateSamlToken(AuthVerifierImpl.java:77)
        at com.vmware.vim.sso.http.impl.AuthVerifierImpl.verifyToken(AuthVerifierImpl.java:66)
        at com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil.verifySecurityHeaderImpl(SsoOverRestVerifierUtil.java:183)
        at com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil.verifySecurityHeader(SsoOverRestVerifierUtil.java:109)
        com.vmware.vcde.common.services.cm.servlet.SsoAuthenticatedFileStreamServlet.doGet(SsoAuthenticatedFileStreamServlet.java:103)
.
.
.
.
2016-01-20T14:18:01.801Z | ERROR    | unset-opId       | tomcat-http--39           | SsoOverRestVerifierUtil        | Failed to verify request signature using following; host:<vCenter FQDN>, port: 443, uri:/cls/resourcebundle
2016-01-20T14:18:01.801Z | ERROR    | unset-opId       | tomcat-http--39           | SsoAuthenticatedFileStreamServlet | doGet: SSO verification failed for client <vCenter IP Address>
com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil$SsoAuthException: com.vmware.vim.sso.http.AuthException: The SAML token is invalid!
        at com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil.verifySecurityHeaderImpl(SsoOverRestVerifierUtil.java:194)

In ovf.log, you are looking for:

ovf.log
-------
2016-01-20T14:18:01.792Z | DEBUG    | unset-opId       | tomcat-http--23           | SsoOverRestVerifierUtil        | Trying to verify request signature using following; host:<vCenter FQDN>, port: 443, uri:/ovf/resourcebundle
2016-01-20T14:18:01.804Z | ERROR    | unset-opId       | tomcat-http--23           | SamlTokenImpl                  | Signature validation failed
javax.xml.crypto.dsig.XMLSignatureException: the keyselector did not find a validation key
        at org.jcp.xml.dsig.internal.dom.DOMXMLSignature$DOMSignatureValue.validate(Unknown Source)
        at org.jcp.xml.dsig.internal.dom.DOMXMLSignature.validate(Unknown Source)
        at com.vmware.identity.token.impl.SamlTokenImpl.validateSignature(SamlTokenImpl.java:653)
        
        
2016-01-20T14:18:01.805Z | ERROR    | unset-opId       | tomcat-http--23           | SsoOverRestVerifierUtil        | Failed to verify request signature using following; host:<vCenter FQDN>, port: 443, uri:/ovf/resourcebundle
2016-01-20T14:18:01.805Z | ERROR    | unset-opId       | tomcat-http--23           | SsoAuthenticatedFileStreamServlet | doGet: SSO verification failed for client <vCenter IP Address>
com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil$SsoAuthException: com.vmware.vim.sso.http.AuthException: The SAML token is invalid!
        at com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil.verifySecurityHeaderImpl(SsoOverRestVerifierUtil.java:194)
        at com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil.verifySecurityHeader(SsoOverRestVerifierUtil.java:109)
        at com.vmware.vcde.common.services.cm.servlet.SsoAuthenticatedFileStreamServlet.doGet(SsoAuthenticatedFileStreamServlet.java:103)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at com.vmware.vcde.common.services.cm.servlet.DispatcherServlet.service(DispatcherServlet.java:53)

In ts.log, you are looking for:

Ts.log
---------
2016-01-20T14:18:01.792Z | DEBUG    | unset-opId       | tomcat-http--14           | SsoAuthenticatedFileStreamServlet | doGet: Entering (/ts/resourcebundle)
2016-01-20T14:18:01.805Z | ERROR    | unset-opId       | tomcat-http--14           | SamlTokenImpl                  | Signature validation failed
javax.xml.crypto.dsig.XMLSignatureException: the keyselector did not find a validation key
        at org.jcp.xml.dsig.internal.dom.DOMXMLSignature$DOMSignatureValue.validate(Unknown Source)
        at org.jcp.xml.dsig.internal.dom.DOMXMLSignature.validate(Unknown Source)
        at com.vmware.identity.token.impl.SamlTokenImpl.validateSignature(SamlTokenImpl.java:653)
2016-01-20T14:18:01.805Z | ERROR    | unset-opId       | tomcat-http--14           | SsoOverRestVerifierUtil        | Failed to verify request signature using following; host:<vCenter FQDN>, port: 443, uri:/ts/resourcebundle
2016-01-20T14:18:01.806Z | ERROR    | unset-opId       | tomcat-http--14           | SsoAuthenticatedFileStreamServlet | doGet: SSO verification failed for client <vCenter IP Address>
com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil$SsoAuthException: com.vmware.vim.sso.http.AuthException: The SAML token is invalid!
        at com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil.verifySecurityHeaderImpl(SsoOverRestVerifierUtil.java:194)
        at com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil.verifySecurityHeader(SsoOverRestVerifierUtil.java:109)
        at com.vmware.vcde.common.services.cm.servlet.SsoAuthenticatedFileStreamServlet.doGet(SsoAuthenticatedFileStreamServlet.java:103)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

Cause:

According to VMware support, these log entries show no security context for the user. Without that Security content the user cannot perform actions on the content library.

Resolution:

We found the signing cert and its root CA used by SSO from vmware-identity-sts.log and took out the ssoserverSign and the root certificate and added them to the CA to TRUSTED_ROOTS using the below mentioned vets command.

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store TRUSTED_ROOTS --alias roo51 --cert 51root.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store TRUSTED_ROOTS --alias roo52 --cert 52root.crt

Then restart all services. Run the following commands. (No there is not a –restart or –reset, use both commands).

service-control --stop --all
service-control --start --all

Thats all for today folks. Hope this helped!

Posted on

Professional Growth: Part 2 & 3 – Base Requirements and Upgrades

Continuing the series on professional growth, which started with my last post: Part 1 – Inspirational Beginnings. I started off asking “How did you get started and how long have you been in the game?” I got some interesting results and this next post I asked the question: “What traits, methods, knowledge or experiences did you find crucial to your growth?” Also updated with question 3, “What has changed since then, what’s new to be done?” Continue reading “Professional Growth: Part 2 & 3 – Base Requirements and Upgrades”