I want to let you know about someone who represents the very best of our community and who needs your help today.
Over the last16 years in working with Microsoft Dynamics, I have been fortunate enough to personally meet thousands of people in the Dynamics 365 and Power Platform community. From partners to customers, developers to low code advocates, they are the most creative, passionate, funny and often wacky individuals. We are all lucky to have a community that is so rich and diverse, and unfailingly generous.
Britta Rekstad, aka @MacGyverCRM, embodies all of the best things about our community. If you don’t know her, let me tell you that she is in fact the only really humble Microsoft MVP you will ever come across. She is not a self-promoter. In fact, she is a promoter of others. I was most impressed when I learned a few years ago how she had given up many lucrative opportunities that came her way as a skilled MVP so that she could focus on mentoring and training women who had no IT background and who had encountered difficult circumstances. Britta was passionate about her mission to help these women build a career and a rewarding life by introducing them to the technology and the community that we all love.
She is not a self-promoter. In fact, she is a promoter of others.
I knew that Britta had been fighting a challenging battle with a rare illness, but until a couple of months ago I did not realize how it had worsened to the point that it has all but taken her out of action in our community and drained away her finances. As a community we are all poorer for it.
She has some close friends who have set up a CaringBridge site where you can learn more, but I want to personally challenge you to take a few minutes, get out your bank card, and donate on her gofundme page. If you have benefited from this amazing community, now is your chance to give back.
File this one away under lessons learned. Recently I was updating a demo environment of Customer Insights that I had originally set up to use one set of enrichment parameters. The original settings used Brands and Categories related to coffee retailers, but for the purposes of a new demo, I wanted to update the environment to see what types of enrichment I could get for financial services and use this information to build new segments. In the Enrichment area, I updated the brands and categories to use some familiar names and topics related to financial services:
After making these changes, I ran the Enrichment again:
However, when I viewed my Customer Cards (profiles) they still showed their affinity for coffee related brands like Starbucks and categories like coffee roasting.
After reaching out to Microsoft, I learned that Customer Insights takes a snapshot of the profile data in order to maintain consistency of the data. To update the snapshot and see the most recent enrichment on the profiles, it was necessary to update the dependent Measures and Segments as well:
It makes sense: Your segments and measures might make use of the enriched data, so you wouldn’t want them to be out of sync after you change the enrichment parameters. Once I updated these, I now was able to see my updated profiles and use this data to make new segments.
A common question that comes up when you begin to discuss Customer Insights and its ability to unify and report on your customer data is: “Why can’t I just use Power BI for that?” I thought it might be helpful to provide some bullet points to help clarify how Customer Insights is differentiated from Power BI.
Artificial Intelligence/Machine Learning: Customer Insights leverages sophisticated algorithms to help match customer records from disparate systems that don’t have IDs or keys in common.
Common Data Service (CDS): Data that you ingest into Customer Insights can be mapped to CDS entities. For example, if you connect a contact from your order processing system, you can map it to the CDS “Contact” entity. Or if you bring in lead data from salesforce.com, you can map it to the CDS “Lead” entity. This is an optional step, but it allows you to begin to normalize your data so it can be used in other applications.
Extensible Machine Learning Models: Once you have unified your data in Customer Insights you can leverage the UI to create measures and KPIs. But you can also extend this functionality by incorporating custom machine learning models by consuming them from Azure Machine Learning Studio.
Third-party Data Enrichment: In addition to unifying the data you own internally, you can overlay third-party data to enrich your customer profiles. Microsoft makes this easy with built-in enrichment functionality from the Microsoft Graph to help you understand your customers’ potential interests and brand affinities.
Segmentation: Customer Insights allows you to create segments (both static and dynamic) that span attributes across your unified and enriched data sources. These segments can be leveraged in your marketing automation system, with built-in connectors to Dynamics or by exporting the segment for use in another application. (In upcoming releases, Microsoft will also have “recommended” segments generated by algorithms that spot patterns and opportunities in your customer data that you might not otherwise look for.
Integration with Power Platform/Dynamics: This may not strictly be a differentiator from Power BI, as of course Power BI also integrates nicely with its family of products. But Customer Insights offers a pre-built solution that can be installed in Dynamics 365 Sales to add customer cards/measures to the Contact forms so you can quickly leverage your unified data. There are also connectors so you can build Power Automate Flows/Automations (do we still call them Flows?), Power Virtual Agents, Power Apps, etc., that leverage your unified customer data.
Time-to-Value: Another huge differentiator between Customer Insights and Power BI (and most other BI tools) is the time-to-value. Connecting to multiple data sources, normalizing them, matching them, and extracting meaningful, actionable insights can be a huge undertaking. Not only does Customer Insights make this process fast and easy, it is UI-driven, meaning you can get to these valuable insights without having to resort to code and complex queries.
Approachability/Relevance to Marketers: While Customer Insights is not intended to be an “end user” application, it is designed so that marketing analysts and professionals will feel comfortable working in it. While both Customer Insights and Power BI may benefit from having a data analyst involved in the initial setup and configuration, I would argue that even a seasoned marketing professional might have difficulty extracting ad hoc data from Power BI. But they can quickly become adept at building segments in Customer Insights for use in their marketing efforts.
Customer Profiles with Demographic, Behavioral Data and Activities: The old saying about “if your only tool is a hammer, every problem looks like a nail” comes to mind. Power BI is powerful and allows you to present data in many ways. But it is a general or horizontal tool for data analysis and reporting. Maybe it is more of a Swiss Army knife as opposed to a hammer. Customer Insights, on the other hand, is focused specifically on understanding your customer and the data points surrounding your customer – how they interact with your website and other marketing channels, what they purchase from you, their interests, etc. For professionals, it is often better to have a tool that is specifically crafted for a specific job, rather than a general-purpose tool that “almost” does the job as well.
Data Ranking: As you unify your customer data, Customer Insights allows you to choose which data source will “win” for any attribute you wish to use on the customer card/profile. For example, your e-commerce data might be more reliable for email addresses, but your CRM data is more reliable for street addresses. Your frequent flyer data might be a more reliable source of mobile phone numbers, but your support desk data might be more reliable when it comes to work phone numbers. As you bring all of this together to make a single view of your customer, you can rank which system will win if both have different data for that attribute. This would be much more difficult to achieve in Power BI (I’m sure one of my Power BI friends will tell me how to do it though!), but it’s a simple matter of point-and-click in CI.
Leapfrogging (Easy Introduction to Sophisticated Data Analysis): Customer Insights makes it easy for businesses to begin using tools and techniques that are typically reserved for very large or sophisticated enterprises. While those organizations who do not leverage CI might have to invest in expensive data scientists and months-long implementations, you can leapfrog them by quickly getting the value of these tools from Customer Insights. For example, connecting your data through the Customer Insights interface will bring your data into an Azure Data Lake. You don’t need to know anything about data lakes, how to construct them or use them, to make use of Customer Insights. But as your organization grows in sophistication, you will have a nicely managed data lake already available to leverage for other applications and scenarios. Similarly, your organization may be years away from building and using your own machine learning models, but with Customer Insights, you can begin using this technology right away and open the door much sooner to the possibilities of ML than you might otherwise have done.
Back in September I had a great conversation with Ulrik “the CRM Chart Guy” Carlsson on his Power BI podcast. We discussed the opportunity around Customer Insights as well as how it fits into the bigger story of the Power Platform. Ulrik is a great interviewer – if you didn’t catch it the first time around, have a listen:
With the 2019 Wave 2 release rolling out from the Microsoft Business Applications Group and all of the surrounding fanfare of announcements and updates, and with Microsoft Ignite coming to a close this week, partners in the Dynamics/Power Platform space have a lot to digest this month. Let me take this moment to help you sort through and prioritize what you decide to focus on with your teams over the coming month or two. Not surprisingly, I’d like to suggest that you put Customer Insights at the top of your list. Let me explain.
One word: Salesforce.
It is no secret that the Microsoft Dynamics 365 product team has been taking direct aim at salesforce.com for years. What might be less-appreciated outside of our industry is just how successful Microsoft has been in this competition. In years past, industry observers might have been justified in saying that salesforce.com was the innovator and leader in the business applications/CRM space, and Microsoft was regularly seen to be the late mover or also-ran, bringing technologies to market often years after salesforce.com had introduced them to their audience. Salesforce.com was the first mover in the SaaS CRM market, at least at scale, around the beginning of the millennium. It would be 8 years before Microsoft was able to launch Microsoft CRM Online, and then only in North America. And think about marketing automation: salesforce.com purchased ExactTarget and created their Marketing Cloud years before Microsoft introduced their own marketing application. Think about AppExchange vs AppSource, Trailhead vs. Microsoft Learn, Lightning vs. Unified Interface. The list could go painfully on.
But Microsoft has been rapidly closing the gap and transforming its ecosystem of business applications in a way that could potentially leapfrog salesforce.com. To this observer, perhaps the first and most significant sign of this change of leadership is Microsoft’s entry into the CDP, or Customer Data Platform, space, with the launch this past April of Customer Insights.
To understand why this is so significant, you have to consider the bigger picture of the CRM and martech space over the last two decades. Back in the previous century, these technologies were the domain of only the largest enterprises. With the rise of SaaS, they became more accessible. Today, as we approach the third decade of this new century, they are table stakes for every business bigger than a mom-and-pop storefront. The affordability and power of CRM and marketing automation means that these technologies are almost ubiquitous, and not having strategies around them almost certainly puts a business at a huge competitive disadvantage.
The promise of these technologies was to help companies to scale their processes – whether sales, service or marketing – to larger and larger audiences of prospects and customers. But at the same time as businesses have been scaling their reach through the use of CRM and martech, a revolution has taken place amongst consumers. Starting with B2C consumers, the purchasing process and decision-making process has become something that is largely completed without the direct input of the business that is trying to sell to them using their highly-scaled CRM and marketing tools. This revolution has been driven by the adoption of smart phones, social media, and ubiquitous internet access. And it has also translated to the B2B space. Business buyers are also individual consumers, and they have taken their shopping habits to work with them. They now conduct their research and get to the point of a purchase decision often well before the seller even knows who they are.
In other words, we may have reached a point where CRM and older martech has scaled a business’s reach as far as it can scale – diminishing returns at best.
The goal of businesses now must be to flip the story on its head: approach their selling and marketing processes from a starting point of the customer. It’s not sufficient any longer to be able to build a subscriber list that you can target with your monthly newsletter. That may well be a necessary ingredient, but it is hardly a sufficient one if the goal is to reach this revolutionary new consumer. In marketing terms, the mandate now must be that businesses adjust their sales and marketing processes around their customer, around a “segment of one,” instead of simply around how they want to improve their processes internally.
That is why over the last few years, savvy marketers have begun to embrace the emerging tech of Customer Data Platforms, CDPs. CDPs are designed to deliver on the promise that CRM once had: get to know your customer at an individual level so they might be more inclined to let you join their buying process.
Why This is Important to Microsoft Partners Now
The reason all of this is so important and timely if you are a Microsoft partner with a focus on traditional CRM (and of course, nowadays, Power Apps and the rest of the Power Platform), is because salesforce.com announced earlier this year that they would be jumping with both feet into the CDP space. After talking CDP down in previous years, they announced at Dreamforce that they would launch a pilot of their Customer 360 CDP application in November 2019, and plan to make it generally available in the beginning of 2020. But there is another “but” to this story: Salesforce’s CEO for marketing, Bob Stutz, departed the company to head back to SAP. (Yes, the very same Bob Stutz who had previously been at Microsoft running Business Applications and buying up a number of not-so-memorable applications like Marketing Pilot.) Salesforce.com announced a new head of their marketing group just last week, and he will be coming into the role at the very same time they are supposed to be launching their CDP pilot.
I see that as an opening for Microsoft and its partners to gain even more ground with Customer Insights before salesforce’s new marketing CEO has a chance to get his feet under him. And marketers are exactly who you should be selling to if you are a Dynamics/Power Platform partner who cut their teeth on CRM.
Follow the steps in this article to learn how to scrape data from websites, structure it in tables, and use it in a Power BI report. In this example, I show how to create a Power BI calendar of events that draws data from two publicly accessible websites.
Download Power BI Desktop (not to be confused with Power BI Report Builder, which is the new name for the tool used to build SQL Server Reporting Services Reports and creates RDL files). To download Power BI Desktop, go to powerbi.microsoft.com and click on the Products menu, or open the Windows Store on your machine.
After connecting, click the button “Add Table Using Examples”
This shows you a preview of the web page where you can begin to select information to place in columns to build a table:
Place your cursor in the first row and begin typing the text that represents an example of the information you want to capture in that column:
You can type a couple of examples if necessary to make sure the Power BI is grabbing the correct info. Tab away and it will fill in the rest of the column.
Rename the column. I like to use a convention that describes the source of the information.
Then tab to create the second column and repeat the step for the next bit of information. Sometimes the webpage will not have the data broken out in a way that makes it easy to parse just the bits that you want. In this example, I just want the date, but I’m getting a long string with the word “When”, a colon and a bunch of empty spaces.
Once you have selected the raw data from the website that you want for your table, click OK.
Tip: You can take a look at the webpage that you are using as a datasource in a browser and use the browser tools to see which elements of the page might be parseable by Power BI. Power BI will look for CSS IDs and selectors to distinguish the elements that make up the page:
Next, Transform your Data
Back on the Navigator screen, click “Transform Data”:
In the Transform window, I am going to use some transformations to get the data I am interested in. For this first column, I will Split the string by the positions of the characters in the string:
When I do that, Power BI looks at the string and automatically shows me where each part of the string starts:
So if we look at that information, we can see that position 0 must be where the word “When:” starts, and position 104 must be where the date starts. 108 must be the day, 111 must be the year, and so on. I don’t want to break the date out into its sub-parts, so I am going to just split it at positions 0 and 104:
This gives me a new column that just starts at position 104: Tip: If I need to undo a step because I mess something up, I can delete it on the right:
Next, I want to just get the date out of my new column. I am going to Split the new column by a delimiter. I don’t want the stuff that starts with the word “from” so I can use that word as a custom delimiter:
Check out what happened here, automatically. After I split that column out, Power BI automatically recognized that this data is a Date type. I can tell by the icon at the top of the new column:
I do a similar “Split by Position” to parse out the location and rename that column. It took some trial and error to find the exact position that the location starts. It turned out that I wanted to split by positions 0 and 117. Then I renamed the column to D365UG_Location:
When I’m satisfied with my table, I click on “Close and Apply”:
Rinse and Repeat
I want to combine this data with another data source, so let’s try another web page. I will repeat the above process with data from https://www.365portal.org/events/.
Now I have two tables with data I can use in my report.
I can continue to add more in the same manner, or even connecting to other types of data. I’ve also gone ahead and connected some data from CDS.
Before I proceed, I am actually going to rename the columns so that across the three different data sources, they are uniform for Name, Date and Location. That will let me use another feature of Power BI to create a new table that combines all the data I scraped from the websites and pulled in from CDS.
I want to combine these different data sources into one new table. So in the Power Query editor, I click on “Append > Append as New” and select the three source tables:
This creates a new table for me that has all the columns from all of my data sources. I have removed the columns that I am not interested in for the next step, so all I am left with is Name, Date and Location. I have also renamed this new table to “Combined”:
Click “Close and Apply” again
Design your report
Now I can begin to design what I want this to look like. I want to add a new control to display this data, so I am going to import it from the Power BI marketplace: I’ve opted to use the MAQ Calendar since it has a decent layout and navigation to see the calendar by day, week or month.
Lastly, I am going to embed this on a portal using a simple iframe:
On my portal website, d365calendar.com, (which runs on Microsoft’s Power Portals by the way!) I’ve added a page and in the HTML, I paste the iframe code from Power BI:
And here is what it looks like live on the site:
These are some of my favorite tricks with Power BI, and I hope you’ll enjoy using them now as well. Leave a comment below and tell me what you think!
A continuation of a series of posts diving into Customer Insights
Looking at a Customer Card or the home screen dashboard in Customer Insights, one of the first things you’ll notice are the tiles that display various numbers, or measures, representing a customer’s interactions with your company. These might include things like a customer’s average purchase when visiting one of your bricks-and-mortar retail establishments, their average online purchase, or lifetime value.
Types of Measures
There are several different types of measures:
Customer Attribute – Configuring this type of measure allows you to add a new attribute for display on the merged contact. For example, if you wish to combine all of an individual contact’s purchases from across various systems into one field to show the individual’s total spend.
Customer Measure – Similar to the customer attribute above, but with an added dimension like time. For example, if you wish to show how many times on average a customer visits your online store, or how much do they spend across all your systems on average per month.
Business Measure – A business measure is a way to look across your entire customer base in aggregate to answer questions at the business level rather than at the individual contact level. This is the type of measure you would configure to answer questions like “what is the average purchase at our retail/POS locations?”
Variables are operations that you want to perform for each customer record in Customer Insights. One of the more interesting capabilities you’ll come across when building a measure is the Variable Expression editor. It is not intuitive at first. On the right, it will show you the tables and attributes from your connected data that you have chosen as you begin building out your measure. I’ve found that if you click the attribute on the right, it will put it in the expression editor for you, saving you from having to figure out the correct syntax to use if you were to type it in manually.
After clicking one of the attributes on the right to put it into the Expression area in the correct syntax, you can simply type the mathematical operator you want to use. You can also group operands using parentheses in order to perform more complex operations. For example, you might want to create an expression like “(Total Web Purchases + Total Store Purchases) / RewardsPoints” to determine how many dollars your customer had to spend across both your website and bricks-and-mortar stores to earn each reward point they’ve been granted. (Perhaps your letting some customers earn those valuable loyalty points too cheaply because they’ve figured out how to game your loyalty system!). That expression would look like this:
After creating a variable for each unified customer record, you’ll need to apply a function to it. Do you want to look at the average result of your variable operation across your entire customer base? Or a sum or count of those results? That is defined in the Function.
Note: A function creates fields where the results of your measure are stored. Behind the scenes, when you define a measure you are creating a new “entity” that will be listed in the Data area of Customer Insights. If you open that entity, you can also view the fields that were created for it as a result of defining the function.
Creating measures can take some experimentation – it’s worth it to begin exploring your data and playing around to see what types of insights these measures can provide your organization. It’s very likely you’ll uncover some things that weren’t obvious before your customer data was nicely unified.
This post is part of an ongoing series on Microsoft Dynamics 365 Customer Insights. Check my blog feed for previous posts on this topic!
After you have connected and unified your data, you can begin to build your customer card and enrich the customer profile using built-in functionality. Let’s take a look at how to do this.
Enrich your Customer Card
Enriching the customer profile takes advantage of data from Microsoft’s Graph, which is basically hundreds of millions of data points that Microsoft has collected and categorized about consumer behavior by three facets: age, gender and location (based on the customer’s postal code). Currently the Graph data set that is available in Customer Insights only covers the United States.
The idea behind enriching your customer profile is to compare your existing customers with the interests and brand affinities of people who are similar to them, based on age, gender and location. For marketers, this is invaluable and very difficult and costly to do with their own data.
Imagine if you are a marketer at a consumer retail business with 10,000 customers: Wouldn’t it be great to be able to look at your customer base and have some idea of how much each customer might like a particular brand (perhaps a competitor’s brand, or one that is similar to your own)? Or if your customer is interested in outdoor sports or fine wine? How would you begin to go about that? There are data sets available online, but you would have to try to slice and dice your own data so that corresponds to the available market study data that is available, and you would of course have to purchase that data. There’s also a good chance that the data you purchase is out of date. But in Customer Insights, Microsoft handles this problem for you with Microsoft Graph.
The first step is to select two of the three facets to compare to your customer data:
In this example, we are looking at our customers based on age and location. Next, we have the option to choose the brands and categories that our customers are likely to have an interest in. We can choose from any of the thousands of brands that the Microsoft Graph data set already contains, or we can let Microsoft choose for us by selecting our industry:
After selecting these parameters, click the Run button at the top, and CI will begin to compare your customer data with the Microsoft Graph data. When it is done, you will see how many customer profiles have been enriched.
Building the Customer Card
If you are following along with previous posts, you’ll already have done the first step in building your Customer Card, which is the “Merge” step of data unification, where you would have selected the fields from your different data sources that you want on the card.
Navigate to the Customers area of CI. Before viewing your Customer Cards, you will be prompted to specify which fields will be used to index your customers so you can search and filter the views:
This process is a quick, wizard-driven step, and, once completed, you will automatically be taken to the view of your customer profiles:
The fields displayed on the Customer Cards are the ones that we specified in the “Merge” step of our data unification. The great thing here is that I may know that we have more reliable birthdate information in our ERP system, but more up-to-date phone numbers in our CRM or loyalty system. Since I defined which field would be the “winner” during the Merge step in case more than one data source has the same information, the Customer Cards show me the most reliable and complete data for each field.
Drilling into each card lets me look at the particular individual, where I can see the fields I have chosen for the merged Contact, their average purchases (or whatever other measures I have configured), and the brands and interests they might enjoy based on their age/location. If you have also connected a data source that represents activities like phone calls, website visits, or emails, the timeline will show you this as well.
Continue to follow this blog series for more on Microsoft Dynamics 365 Customer Insights and other Dynamics technologies. In the coming weeks we’ll have a look at how to build measures in Customer Insights, manage your connected data, build segments, and more. Leave a comment with your questions, ideas and your own tips!
Follow this series to explore Dynamics 365 Customer Insights. For an overview of Customer Insights, read the first post in the series here. The second post covers the topic of getting started and connecting your data to CI and can be found here.
After bringing in your data and making sure that the types of data have been correctly categorized, the next step is to unify all your data sources. To do this, you need to tell Customer Insights what represents a customer across different systems, how they might be matched up, and then create a “merged” customer profile from your source data. Unifying your data consists of three steps:
Map – Map your ingested data to a common model (CDS entity, for example)
Match – Determine what consists of a match from one data source to another
Merge – Create a unified “customer” entity for use in CI
Let’s consider a scenario where we have brought in data from four different data sources. Two of them have the idea of a contact – one is an eCommerce system that has a Contact entity, and the other is a Loyalty Card tracking system which refers to them as customers. The other two data sources have data about purchases and activities.
I want to determine how many unique people we interact with across the two systems that have data that represents people, and find out how many of them are in both systems. Then I will use that data to understand them more deeply.
I’ll start by going to “Map” under the Unify area and select the two tables that represent people:
Mapping at this point is a simple exercise. The system is intelligent enough to make some assumptions about the data types that have been brought over. For example, it will automatically assign model types to the DateOfBirth field of Person.BirthDate and the PostCode field is identified as a Location.PostalCode type. The main thing I need to do is to let CI know which field from each source represents the primary key:
In the matching step, start with the data source that represents the most complete and accurate representation of your customers. In this example, our eCommerce system has the most customer information, and the loyalty program data source is more of a subset. We’ll include all the records from both systems so we have a union of both data sources to work with.
If you have a third or fourth dataset that represents customers/contacts, you can add these. CI will first check for matches between the Primary table and Entity2 and come up with a list of matches. It will then take that list and check for matches in Entity 3. From the resulting list it will then check for further matches in Entity 4, and so on.
After establishing that hierarchy, we
want to specify a rule to match each customer that appears in both systems. You
can match based on one or more conditions. For example, we want to look at the
full name and email across both systems to find matches. You have some options
for each condition:
Precision: Exact will only match precise matches, high/medium/low give you increasing levels of fuzzy matches. If you are going to create a segment for a marketing list, you might choose a lower match precision where customer records from the two systems are matched with a lower level of confidence, but if you are going to run financial analyses, you might choose a higher precision.
Normalize: The normalize option can be used to ignore whitespace, capitalization, numerals, or to only match if the fields are of the same attribute type
After creating the rule, it will run against your data and you can see a preview of the different matches based on the confidence level that CI has in the match it has determined.
The last step in unifying your data is to create a “merged” Contact record in CI. The purpose of merging is to create the meta-customer record that you will see in CI and to instruct CI on how to reconcile conflicting matches. This merged contact is represented in CI as a “Customer Card”, also referred to as a profile. A merged attribute is an attribute that exists in more than one data source and represents the same piece of data. For example, both of our data sources have a FirstName attribute, but the rank shows that we will give preference to the FirstName field from our eCommerce data source in case they differ.
The end result of mapping, matching and merging will be a Customer Card representing each person’s profile in Customer Insights. In the next article, we’ll take a look at the Customer Cards and some of the other ways you can configure Customer Insights to gain deeper understanding of your customers.