From time to time, and for whatever reason…. like oh…say… your solution contains multiple projects and you decided to upgrade the cloud service project to the latest Azure SDK from version x and for whatever reason you have a reference to a specific Azure SDK component in one or more projects that are not your Web / Worker Role project… Guess what… you might run into this (Role perma-recycling fun)  after deploying to Azure that when tested on the development fabric it is working just fine.

*Digressing*

You know what, sometimes Azure baffles me.  I upgraded my Azure SDK previously and managed to deploy it to the cloud and have it running just fine on one deployment and then when doing another deployment some time after it just refuses to start and start the recycling crap… and this is not a staging slot swap to prod slot thing, it’s direct deployment onto the production slot too.  I haven’t deep dive into Azure internal to understand what’s going on when you do deployment, but I think there are optimization steps (to make the deployment process faster perhaps) where sometimes the same VM is being used and when previously it still have the residual DLLs from previous deployment and when this happens, everything work just fine even after the SDK update and you forgot to update the outdated DLL… and sometimes, Azure just says, “… meh… screw you… I’ll drop your bit on a totally new VM and  guess what, HAHA!!, the bits that you think will be there is no longer there, so there…” and the recycling crap happens.

Is this what happened? I don’t know… maybe.  Some insights into this perhaps or not… http://blog.smarx.com/posts/what-happens-when-you-deploy-on-windows-azure.  Step 2 is a bit ambiguous…  Maybe Azure Friday will invite someone to explain how this thing works internally? *Nudge nudge wink wink*

*More digression*

You know what… this reminds me of something else… Sometimes when my Continuous Deployment build got processed by Visual Studio Online, I get this annoying Cannot copied such and such blah error during the build and the build crapped and stopped there. FAIL!  And the only thing I need to do is re-queue the build and guess what… It successfully build and deploy…

What the !(*&$&!#!!!  My theory… there is a build agent machine that is not configured right… and I was just unlucky to get sent to that agent from time to time.

*End of more digression*

Geez… what is this… Digression Inception?

*End Digression*

If you are a newb at Azure like I was previously, you’ll be scratching your head for a looong looong time trying to figure out what the heck is going on… 
It worked on my machine… Damn it..
*pull hair* … oh no *bald spot forming*.

After running into this issue a couple of time and doing searches for answers on stack overflow and what not… You’ll finally realize that you have missing reference and the web / worker role just can’t start your application because of that, which I think is the most likely candidate anyhow…

So, how do you go about troubleshooting this?

Remote Desktop is Your Friend

One easy way to do this is via Remote Desktop.

You DID configure remote desktop on that role, did you not…?

No?  You dummy you, go do it now!

… … …

You DO know how to configure remote desktop for your role, do you not?
No? *sigh*  Read this.
I’d recommend creating your own client certificate to use for this and CHECK IT INTO YOUR SOURCE REPOSITORY.  You never know when you need the certificate again… for example, when you need to redeploy the cloud service to a different Azure subscription and the original certificate is now nowhere to be found for reasons like the original developer who created the certificate on his / her machine left the company, the machine got wiped and YAY… no more certificate copy anywhere…  Happened… True story.

So, now that’s out of the way, go launch remote desktop and login…

What? You don’t remember what’s the password for the VM? Are you kidding me? *sigh*

Go read this and follow the instruction on how to reconfigure…

Event Viewer is Also Your Friend

Okay… now that you are logged into the remote desktop, go start Event Viewer.  If you don’t know how to do this, go quit and sell ice cream in a truck… or ask your boss to hire a DevOps... or go learn how to do it!!  Google or Bing maybe… like “How to launch Event Viewer from Windows Server 2012 R2?”…

Got Event Viewer up?  Good.

Now go open the Windows Logs node and drill into the Application node and find some errors (such as one that is shown below…)

AHA! See that FileNotFoundException!!! I knew it, File Not Found…. It’s the same thing that is shown in the Azure portal.

Yeah…. but… what file?…. Uhm….
What a freaking useless error message.

Okay… maybe we are looking at the wrong place…. Let’s see what else is there…

Let’s try the Applications and Services Logs node… drill drill, aha… Windows Azure/Diagnostics/Bootstrapper node… It’s related to starting up the role right?  Bootstrapping = start up, right? Must be here..

AHA! I see some errors.

… WT #(@$! CRuntimeClient: OnRoleStatusCallback #(*@#(*@#(*$  What the heck does that mean?

Yet more useless crap!

*sigh*  Oh well, let’s continue mining…

Applications and Services Logs/Windows Azure… WELL… what do you know… More errors… and this time.. we found GOLD… well, sort of…!

Could not load file or assembly ‘Microsoft.WindowsAzure.ServiceRuntime, Version=2.3.0.0…’ at XXXXXXX.Shared.Configuration….

Now we are going somewhere…

What’s Next?

Well, now that you know what it can’t find during the startup, you just need to ensure that the DLL or whatever component the role needs to start is correctly packaged and deployed to Azure.  Upgrade the DLL using Nuget and fix your references, and make sure to set the missing reference to be copied always to the bin folder.

Check in your changes, trigger your Continuous Delivery build or package it and redeploy manually.  Whatever your deployment style is… Hopefully it will work this time… 

If it doesn’t…,
Well, go figure it out. Gave you a clue on how to troubleshoot this… Do I have to hold you hand through all the process? What are you? Kindergarten student? PHAW!

I'm not sure if this only happens on Windows 7 or not, but I encountered such message when trying to start a game after rebuilding my machine using Windows 7 64 bit. Actually, I encountered it pretty much on any game I tried to start that uses DirectX 3D.  Looking around the web only gave me bits and pieces of clues that didn't quite go anywhere.

Some said to copy the offending DLL into %SystemRoot%\System32 directory, which didn't work.  Some others suggested reinstalling DirectX 9.0c, which I also tried but didn't work either; however, this is one step toward the right solution, just not the whole solution.  I tried reinstalling DirectX multiple times using the DirectX End-User Runtime Web Installer and Redistributable ones.  All ended up in error.  The installer suggested that I looked into the log files in the Windows directory called DirectX.log and DXError.log.  It turned out to be an access denied error somewhere along the way during the installation which I found in the DXError.log file.    I then tried to use elevated privilege (Run as Administrator) to run the installer again, but that also didn't work and ended up with the same error.  *scratch head*.

Running out of idea of what to do, I went back to the web to try and find out what could be causing this error.  After lots of time browsing and searching around the web I ended up in a forum post which suggested that I turned off the real time protection of the Forefront client (the antivirus software that I was using) before retrying the DirectX installation again.  I tried that and WOOT... that fixed the problem.  DirectX successfully installed and when I tried to run the games that didn't work before, it runs like cutting butter with a hot knife :) (or something like that)

So, moral of the story... if anything else fail, check your antivirus :).  I ran into similar problems on other software (not game related).  For example, when trying to use port 6666 as K2 Blackpearl Server port, I was blocked by McAfee because it was assuming that was an IRC port that is dangerous to expose in a corporate environment.   A different time, my mail won't go through the SMTP server because...again... it was blocked by the antivirus.

I supposed this is one thing that can be added to List of Dumb Things to Check.

Hope this post helped somebody out there that is banging their head because of lack of answer to similar question.