Using Pymongo Update to perform an Insert and Update

Was working on a module that would iterate through a row list that came back from a query and attempt to insert or update the specific columns of the returned rows to a collection on Mongo. Originally I was planning to use count({"id":row[id]}) > 0 to check for the existence of the given record in Mongo, and then execute a insert of push as needed. However I stumbled the update method which is part of pymongos collection level operators that allows me to execute all three actions, check for existence, insert or push, with one call.

So my original code to insert or update was as shown below:
 #check if row exist
 if coll.count({"MemberId": row["idclient"]}) > 0:
 #Insert row into the Collection
 coll.insert(details) else:
 #update the existing row
 coll.push({"MemberId": str(row["idclient"])}, { $push: { "field1": str(row["field1"]), "field2": str(row["field2"])} })
 As observable from above the push call requires us to decompose the details dictionary object into its elements to update.
However with the push I can skip these three lines and execute the same operation with one call: coll.update(key, details, True)
 I have to set up what key to use and also have to set the upsert flag to be true, then an update() (upsert) operation is performed and any existing document with that "_id" is overwritten. Otherwise an insert() operation is performed.
 See below for the link on stack overflow that provided the motivation for this and the documentation on MongoDB.org that explain the workings of the update method.

http://stackoverflow.com/questions/2801008/mongodb-insert-if-not-exists

http://api.mongodb.org/python/current/api/pymongo/collection.html

jquery.jqplot

Needed to provided graphing and visualization capability to a quick dashboard I am developing; after some experimentation with a few jquery libraries I found jquery.jqplot to have the shortest learning curve. It provided me with the capability to draw out most standard graphs and visual aids. jquery.jqplot also provides the ability to highlight data points on the graph. Which would allow developers to enable selection of and further analysis of a given data point. In the days to come I will be diving deeper into the functionality provided by jquery.jqplot to validate if it will allow me to visualize all the types of data analysis that I need to provide. One interesting I have noticed about jquery.jqplot is that burying the div where the graph is stored in other div's renders the aforementioned div inaccessible when a given data point has been highlighted.

Using SkelJS

I had to build a dashboard for reviewing patient details, very quickly. However wanting to still build something that was clean, fast and which could eventually be ported to my long term solution; I decided to build the UI layer in pure html/css/js. So being lazy I decided to look for prefabricated HTML5 templates. I found one on html5up.net which seemed to serve my purpose of simplicity and having a clean look. After downloading it and starting to play with it I found that it was using SkelJS, not knowing the innards of SkelJS, I decided to forge forward and skin the CSS to suit my needs, the initial skinning was relatively painless, however that impression changed as soon as I got to the re-sizing piece (I needed the left nav bar to not collapse and I needed the div containing the left nav bar and the right viewing container to expand vertically as I added more items to the right container) I feel like I had gotten trapped in a spiders web, SkelJS does it configuration via a combination of configuration setting and JS that resides in the JS files. Removing an innocent looking configuration easily had far reaching effects, also try to follow the code in the skel.min.js file that comes out of the box was a challenge in patience. I eventually managed to resolve my issues, but I don't think I will be using SkelJS in the final solution that I plan to develop with webpy.

Common issues when migrating from 3.5 to 4.0

I recently had to migrate one of our enterprise level applications to .net 4.0 and it was a challenge to say the least in terms of isolating fixes for some of the obscure issues. Here is a list of some of the issues that I face while migrating and how I overcame them:

Problem) There is a duplicate 'system.web.extensions/scripting/scriptResourceHandler' section defined

Solution) In .net 4.0 the config sections scriptResourceHandler, jsonSerialization, profileService, authenticationService, roleService amongst others are defined in the machine.config. So there is no longer a need to explicitly define them. Of course if you need to change any of these settings then you will need to remove the config sections http://msdn.microsoft.com/en-us/library/ms228258.aspx and then readd the config section and modify the settings as desired.

Problem) ‘Schema specified is not valid’ error message gets thrown while trying to upgrade your Entity Model to 4.0.

Solution) Regenerate the connection string of your model. It should look like some below. As you can see, the upgraded version of EF required much more info about where to get its metadata resource.

add name="TestEntities" connectionString="metadata=res://*/BusinessRuleEntity.csdl|res://*/BusinessRuleEntity.ssdl|res://*/BusinessRuleEntity.msl;provider=System.Data.SqlClient;provider connection string="Data Source=(local)\sqldefault;Initial Catalog=TestDB;User ID=TestUser;Password=*****"" providerName="System.Data.EntityClient"

Problem) A potentially dangerous Request.Form value was detected from the client when loading an html page.

Solution) This issue can be fixed by adding the directive to the section of the web.config, and .NET then honours the directive that's in the same root web.config.

Windows Azure Performance Monitoring and Logging

What is managed code?

I have been out for the last year researching and developing on Azure. Now that my Azure project is in a stable state I have more time to start posting on articles I find useful again. Recently I have been on a hiring spree, and have been looking into questions to ask to gauge the true understanding and depth of a potential candidate, and one fundamental question I find that most folks lack an understanding for is what is managed code. So here I have posted a indepth answer to that question.

Managed code is code that has its execution managed by the .NET Framework Common Language Runtime. It refers to a contract of cooperation between natively executing code and the runtime. This contract specifies that at any point of execution, the runtime may stop an executing CPU and retrieve information specific to the current CPU instruction address. Information that must be query-able generally pertains to runtime state, such as register or stack memory contents.

The necessary information is encoded in an Intermediate Language (IL) and associated metadata, or symbolic information that describes all of the entry points and the constructs exposed in the IL (e.g., methods, properties) and their characteristics. The Common Language Infrastructure (CLI) Standard (which the CLR is the primary commercial implementation) describes how the information is to be encoded, and programming languages that target the runtime emit the correct encoding. All a developer has to know is that any of the languages that target the runtime produce managed code emitted as PE files that contain IL and metadata. And there are many such languages to choose from, since there are nearly 20 different languages provided by third parties – everything from COBOL to Camel – in addition to C#, J#, VB .Net, Jscript .Net, and C++ from Microsoft.

Before the code is run, the IL is compiled into native executable code. And, since this compilation happens by the managed execution environment (or, more correctly, by a runtime-aware compiler that knows how to target the managed execution environment), the managed execution environment can make guarantees about what the code is going to do. It can insert traps and appropriate garbage collection hooks, exception handling, type safety, array bounds and index checking, and so forth. For example, such a compiler makes sure to lay out stack frames and everything just right so that the garbage collector can run in the background on a separate thread, constantly walking the active call stack, finding all the roots, chasing down all the live objects. In addition because the IL has a notion of type safety the execution engine will maintain the guarantee of type safety eliminating a whole class of programming mistakes that often lead to security holes.

Contrast this to the unmanaged world: Unmanaged executable files are basically a binary image, x86 code, loaded into memory. The program counter gets put there and that’s the last the OS knows. There are protections in place around memory management and port I/O and so forth, but the system doesn’t actually know what the application is doing. Therefore, it can’t make any guarantees about what happens when the application runs.

How to reverse contents of a file in windows

I recently needed to mimic the linux functionality of being able to reverse the contents of any file, on windows. After playing with a few options, I found vb-scripting to be the least painful way to do this. See below for details:

Step 1:
Save the script below as a .vbs file for example ReverseStream.vbs

' ------- BEGIN CALLOUT A -------Dim Stack: Set Stack = CreateObject("System.Collections.Stack")' ------- END CALLOUT A -------
' ------- BEGIN CALLOUT B -------Do While Not WScript.StdIn.AtEndofStream Stack.Push WScript.StdIn.ReadLineLoop' ------- END CALLOUT B -------
' ------- BEGIN CALLOUT C -------WScript.StdOut.WriteLine Join(Stack.ToArray, vbCrLf)' ------- END CALLOUT C -------

Step 2:
To run the script use the following command on the cmd shell

c:\>cscript c:\ReverseStream.vbs c:\Reversefile.xml