Tuesday, April 15, 2014

Large Lists, BCS, Excel REST, JSOM, CSVs and Office 365 OH MY!

An interesting challenge came across my desk last week:
"How can we import a CSV into Office 365 and use that data to tag other items within the site.  The CSV currently has ~7600 rows and is expected to grow.  We'll also need to re-import the CSV on an ad-hoc basis when the data changes."
The last part of that was the real issue.  There was simply no easy way of doing that.  After trying a few things, falling flat on my face, I'm prepared to share my exploration into the different options.

Each have their own merits and pitfalls.  This post will examine each and try to shed some light on the pitfalls I've found using each of these.


Wiring up an ECT on Office 365 can be a little finicky. I initially had some issues due to 2 BDC Models that were created for the same ECT.  After calling in the eagle-eyed BCS guru, +Fabian Williams, I got squared away.

Immediately after that, I could tell that I was not going to be able to use BCS for what I needed.  In Office 365, there is a hard limit of 2000 items that can be retrieved.  Ironically, the error message that is displayed is not a supported cmdlet for Office 365.

Office 365 BCS Throttle Error
Adding insult to injury, I decided to run a simple test using JSOM.  I wanted to clarify if JSOM would provide me PagingInfo with a BCS List.  Using the code below, you'd expect line 68 to produce a value instead of nothing at all.

Since BCS will not work and due to the limitation of the API, I had to look for another solution.

Custom List

Using a custom list initially worked great.  I'm able to use JSOM, query the list for 5000 items per trip to the server, AND get PagingInfo.  Using the code below works great for this scenario.

Importing the Excel into Office 365 is relatively straightforward and will work for most needs.  The file I used had about ~7600 rows of data.  After importing the file, I noticed the Server Resource Quota was tapped.

Office 365 Server Utilization
So using this approach has 2 problems.  I will not be able to do a mass import again of my data (the list already exists) and the Server Resource Quota points are exhausted.

Excel REST

This seemed like a cool way of getting around the limitations above, so I dove in to find out if this will work for my needs.  After all, I'm allowed to have a *lot* of rows in Excel and I'll be able to easily update the file, since it's in a document library.  Using the code below, I ran into a showstopper though.

There is a hard limit in the API set at 500 rows.  That would be painfully slow to get all of the items or even worse; a user may try to use the form control while this is still querying for data.

Excel REST API - 500 row limit
So that leaves us with our raw data that was exported from SQL and given to us to use.


Updating the CSV will be easy, since it will be stored in a document library.  Now all we need to be able to do is make sense of it.  Using the code below, I'm able to parse the CSV and create an array of objects that I need to pass off to another library.  Also note the use of localStorage.  This is a nice way to cache the data and prevent the retrieval/processing of the data client-side on every page load.  If the CSV is updated, simply clear the browser cache and you'll get the latest and greatest.


All approaches have their merits and pitfalls... BCS and PagingInfo, I'm looking at you!  If the ad-hoc mass-import wasn't needed, then using list driven data would have been my choice.  If I used that approach, I would have still used localStorage though.  It makes sense to cache the processed data since it'll not change very much.  Since my solution works client-side, I'll have to take into consideration the amount of time this takes to render.  I'm getting good performance out of the CSV approach, so I'm going to stick with it for the time being.

Friday, January 10, 2014

Fire Workflows with Initiation Parameters using #SPServices

Firing Workflows using Javascript, I've never had to pass in Initiation Parameters.  This post takes a look at how to do just that and provides some code that will allow easy use of Workflows in Javascript.

Setup Workflow

The workflow has to be set for Manual starting, otherwise this will not work.  Also, to pass in parameters to the workflow, you'll need to have Initiation Variable(s) within the workflow. I've only fiddled with Number and Single Line of Text fields, so if you use other column types, feel free to share your experience.

Workflow Initiation Parameters

This workflow is simply logging the variables out to the Workflow History.  Easy peazy...

Workflow Parameters

Looking over the documentation for SPServices and StartWorkflow, I found some examples that were a great starting point.  After fiddling a bit with 1 field, I decided to test this a little more.  I created a column with spaces in the Name field *gasp*.  I only did this to see how to handle this programmatically, so a word to the wise: Friends don't let friends create columns with spaces...

Reading over the examples, if you have multiple parameters, it says you have to change from passing the column name to this weird pattern: 

<Data><Parameter1>" + parameter1 + "</Parameter1><Parameter2>" + parameter2 + "</Parameter2></Data>

I've found this to not work at all for me at all [sad_panda]... Back to the drawing board, I guess. Then an idea came to me.  Since I'm targeting a column with a space in it, I tried what normally happens to spaces in Static Names: _x0020_.  So, I tried this next:

<Data><TextField>Will it blend?</TextField><With_x0020_Spaces>42</With_x0020_Spaces></Data>

However, this didn't work either!  Very curious to find a resolution, I set out to find out why this didn't work...  Using SPD (SharePoint Designer), you are able to view the files generated by the workflow. Opening up the XML file as text, you can clearly see that SPD removed the space in the Static Name.

Workflow wfconfig.xml
Within this file, all of the Initiation Parameters are visible and it's now easy to tell what's exactly going on.

Workflow Parameter Names
For all of this to work while using multiple parameters, you have to use the exact Static Name as defined in the XML.  The workflow parameters below work just fine for me now.

<Data><TextField>Will it blend?</TextField><WithSpaces>42</WithSpaces></Data>

Code to Fire the Workflow

This function will handle the pain of getting a workflow to fire.  All you need to know is the correct URL, the workflow name, and the workflow parameters ( if any ).

*** Update *** I took my original idea and made it more or less a plug-in for SPServices. Add this function to the SPServices source and it'll work without any issues. Original function:

Friday, November 22, 2013

Use Server Variables In XSLT To Find The Root Site

There’s no way to tell what the site’s Root Web URL will be, nor the protocol used on any given site, so here’s a quick rundown of how to get the Root Web URL using XSLT:

Variable Setup

This example requires these two server variables.

ServerVariable - SERVER_NAME
Here’s a screenshot of what it looks like in SPD.  With these in place we'll be able to determine the Root Web URL dynamically, so this could be used on any site and the code will know exactly what the URL of the root web is.


I've commented the key parts to this simple example, so if you want to just take those pieces you can.  Doing this through the GUI, it adds two <ParameterBinding />’s to your code.  Later in the code, you’ll see two variables that are used:
  • httpProtocol
  • sitePath
These variables are put together to form the URL of the Root Web.

<WebPartPages:DataFormWebPart runat="server" Description="" PartOrder="2" HelpLink="" AllowRemove="True" IsVisible="True" AllowHide="True" UseSQLDataSourcePaging="True" ExportControlledProperties="True" DataSourceID="" Title="" ViewFlag="8" NoDefaultStyle="TRUE" AllowConnect="True" FrameState="Normal" PageSize="-1" PartImageLarge="" AsyncRefresh="True" ExportMode="All" Dir="Default" DetailLink="" ShowWithSampleData="True" ListId="" ListName="" FrameType="None" PartImageSmall="" IsIncluded="True" SuppressWebPartChrome="False" AllowEdit="True" ManualRefresh="False" ChromeType="None" AutoRefresh="False" AutoRefreshInterval="60" AllowMinimize="True" ViewContentTypeId="" InitialAsyncDataFetch="False" MissingAssembly="Cannot import this Web Part." HelpMode="Modeless" ID="" ConnectionID="00000000-0000-0000-0000-000000000000" AllowZoneChange="True" IsIncludedFilter="" __MarkupType="vsattributemarkup" __WebPartId="" __AllowXSLTEditing="true" WebPart="true" Height="" Width=""><ParameterBindings>
                        <ParameterBinding Name="dvt_apos" Location="Postback;Connection"/>
                        <ParameterBinding Name="ManualRefresh" Location="WPProperty[ManualRefresh]"/>
                        <ParameterBinding Name="UserID" Location="CAMLVariable" DefaultValue="CurrentUserName"/>
                        <ParameterBinding Name="Today" Location="CAMLVariable" DefaultValue="CurrentDate"/>
                        <!-- Set these two variables up within your DVWP/XLV Web Part -->
                        <ParameterBinding Name="HTTPS" Location="ServerVariable(HTTPS)" DefaultValue=""/>
                        <ParameterBinding Name="SERVER_NAME" Location="ServerVariable(SERVER_NAME)" DefaultValue=""/>
    <xsl:stylesheet xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata" version="1.0" exclude-result-prefixes="xsl msxsl ddwrt" xmlns:ddwrt="http://schemas.microsoft.com/WebParts/v2/DataView/runtime" xmlns:asp="http://schemas.microsoft.com/ASPNET/20" xmlns:__designer="http://schemas.microsoft.com/WebParts/v2/DataView/designer" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:SharePoint="Microsoft.SharePoint.WebControls" xmlns:ddwrt2="urn:frontpage:internal">
        <xsl:output method="html" indent="no"/>
        <xsl:decimal-format NaN=""/>
        <xsl:param name="dvt_apos">&apos;</xsl:param>
        <xsl:param name="ManualRefresh"></xsl:param>
        <xsl:param name="webUrl" />
        <xsl:param name="HTTPS" />
        <xsl:param name="SERVER_NAME" />
        <xsl:variable name="dvt_1_automode">0</xsl:variable>
        <xsl:template match="/" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata" xmlns:asp="http://schemas.microsoft.com/ASPNET/20" xmlns:__designer="http://schemas.microsoft.com/WebParts/v2/DataView/designer" xmlns:SharePoint="Microsoft.SharePoint.WebControls">
            <xsl:call-template name="dvt_1"/>
        <xsl:template name="dvt_1">                       <xsl:variable name="Rows" select="/dsQueryResponse/Rows/Row"/>
            <xsl:variable name="dvt_RowCount" select="count($Rows)"/>
            <xsl:variable name="IsEmpty" select="$dvt_RowCount = 0" />
            <xsl:variable name="dvt_IsEmpty" select="$dvt_RowCount = 0"/>
            <!-- Use this variable to determine the protocol used by the site -->
            <xsl:variable name="httpProtocol">
                    <xsl:when test="normalize-space($HTTPS) = 'on'">https://</xsl:when>
            <!-- This variable concats all of these variables together -->
            <!-- Example sitePath output: http://iOnline247.me/ -->
            <xsl:variable name="sitePath">
                <xsl:value-of select="string(normalize-space(concat(concat(normalize-space($httpProtocol), normalize-space($SERVER_NAME)), '/')))" />
                    <!-- {...SNIP...} -->

The end result is a nice URL.  Here’s a sample output: http://iOnline247.me/
I'm not sure why the protocol isn't straightforward to get.  I may have missed something obvious, but I didn't see any other way of determining the protocol.  If there's an easier way, feel free to let me know.

Monday, November 18, 2013

Add SublimeText to Windows Context Menu

Since I've made the switch, I've loved SublimeText.  It is by far superior to Notepad++.  There are some small things that I'm still working out, but I'm sure with time they'll go away.  One of those things was the context menu was missing.  I'm sure this was due to me using the portable installer, but it's a must for me to have!  Doing a simple search on the web, I came across this gist that does exactly what I needed:

Here's the code in its entirety... You just never know with the web. ;)

@echo off
SET st2Path=C:\Program Files\Sublime Text 2\sublime_text.exe
rem add it for all file types
@reg add "HKEY_CLASSES_ROOT\*\shell\Open with Sublime Text 2"         /t REG_SZ /v "" /d "Open with Sublime Text 2"   /f
@reg add "HKEY_CLASSES_ROOT\*\shell\Open with Sublime Text 2"         /t REG_EXPAND_SZ /v "Icon" /d "%st2Path%,0" /f
@reg add "HKEY_CLASSES_ROOT\*\shell\Open with Sublime Text 2\command" /t REG_SZ /v "" /d "%st2Path% \"%%1\"" /f
rem add it for folders
@reg add "HKEY_CLASSES_ROOT\Folder\shell\Open with Sublime Text 2"         /t REG_SZ /v "" /d "Open with Sublime Text 2"   /f
@reg add "HKEY_CLASSES_ROOT\Folder\shell\Open with Sublime Text 2"         /t REG_EXPAND_SZ /v "Icon" /d "%st2Path%,0" /f
@reg add "HKEY_CLASSES_ROOT\Folder\shell\Open with Sublime Text 2\command" /t REG_SZ /v "" /d "%st2Path% \"%%1\"" /f

Tuesday, October 1, 2013

Simple[r] jQuery Content Filter

A post titled: Simple jQuery Content Filter for Office 365 Public Website by Doug Hemminger came by my twitter stream that caught my eye.  A real quick win and all around goodness for everyone.  Reading through, I noticed some things I could tidy up.  So I asked him if he'd be okay if I re-factor the code that he blogged about and he said absolutely!


It's almost expected these days to be able to click the text next to a checkbox and have it just work.  Doug is using <span> for the text, so changing this to use <label> and wrap the <input> is a real quick win.

As I was writing this up, I also had another idea...  Why not have all of the filters show no matter how far down you scroll?  This is also a quick and easy win, so it's in there too.


I'm totally guilty of this, but I try to do my best every time.  I checked the source of the page where the code was added and found:
        $(document).ready(function () {
            function SPSToggleView() {
                if (!$("#SPSBusinessCheckbox").prop("checked")) {
                } else {

                if (!$("#SPSInformationWorkerCheckbox").prop("checked")) {
                } else {

                if (!$("#SPSCertificationCheckbox").prop("checked")) {
                } else {

                if (!$("#SPSBusinessIntelligenceCheckbox").prop("checked")) {
                } else {

                if (!$("#SPSDeveloperCheckbox").prop("checked")) {
                } else {
                if (!$("#SPSITProCheckbox").prop("checked")) {
                } else {
                if (!$("#SPSCloudCheckbox").prop("checked")) {
                } else {
                if (!$("#SPSSocialCheckbox").prop("checked")) {
                } else {
                if (!$("#SPSGeneralCheckbox").prop("checked")) {
                } else {
            $("#SPSInformationWorkerCheckbox").attr("checked", true);
            $("#SPSBusinessCheckbox").attr("checked", true);
            $("#SPSCertificationCheckbox").attr("checked", true);
            $("#SPSBusinessIntelligenceCheckbox").attr("checked", true);
            $("#SPSDeveloperCheckbox").attr("checked", true);
            $("#SPSITProCheckbox").attr("checked", true);
            $("#SPSCloudCheckbox").attr("checked", true);
            $("#SPSSocialCheckbox").attr("checked", true);
            $("#SPSGeneralCheckbox").attr("checked", true);


Even though this is a simple solution, I knew it could be made simpler. :) I got it down to this:

$(document).ready(function () {
        var $wrapper = $("#wrapper");

        $wrapper.on("change", "input[data-filter]", function (event) {
            var $this = $(this),
                filter = $this.data("filter");

            $wrapper.find("div[data-filter='" + filter + "']").slideToggle();

The magic is done by hiding the value(s) we are going to hide/show directly onto the <input> as a `data-filter` attribute.  When the change event is fired, that value is surfaced.  Then a simple query of the DOM to find the correct <div>'s that this filter relates to.  The jQuery#slideToggle method knows whether or not the elements are hidden or displayed, so there's no need to check the state of the <input>.