A couple of people were asking for the code to my demos, so here it is! I’m utterly exhausted, but it was such an honor presenting to so many people. My favorite part was getting to thank a hard-of-hearing gentleman in sign language for asking a question at the very end.
A recent viewer of my new Pluralsight course had a question about data gateways and Power BI Premium. Specifically, do you need a pro license to install and administer data gateways? The short answer is probably not!
Installing data gateways
So when you install a data gateway, you need to log in as a user to register it with your tenant. Well it turns out that whoever is used there is set as the default admin for that gateway. I created a user with just a power BI free license, and I was able to install and administer that gateway just fine. I was also able to assign it to other gateways that already existed.
So, for normal usage you don’t have to be licensed with pro to setup and configure data gateways. I was honestly a bit surprised by this, but in retrospect is makes sense. Pro licensing is all about consuming reports.
What about Premium?
So, the original question was about Power BI Premium. Unfortunately, there’s no developer tier for me to test on, but I have a few guesses.
First, I reviewed the white paper and the distinction it makes between pro users and infrequent users is about producing versus consuming reports. It doesn’t really talk much about administration from what I could tell. Same thing for the faq:
Do I need Power BI Pro to use Power BI Premium?
Yes. Power BI Pro is required to publish reports, share dashboards, collaborate with colleagues in workspaces and engage in other related activities.
Next, I did some searching, and found a page about capacity admins, but that doesn’t relate to data gateways specifically.
So based on what I found, I would assume that you don’t need a pro license to manage data gateways for premium. I would assume it would be a similar experience to normal Power BI.
I just gave a presentation for the Excel BI virtual group on database theory, and I’m really happy with how it went. I think it’s an undeserved topic quite honestly. So many people in the excel world learn everything ad-hoc and never have a chance to learn some of the fundamentals.
A number of questions came up relating to the engine and how the performance works. If you are interested in more detail on that, I suggest checking out my talk on DAX.
Here are the slides for my talk:
Video is coming soon as well.
- So, I think that DAX is a pain in the butt to use and to learn. I talk about that in my intro to DAX presentation, but I think it boils down to the fact that you need a bunch of mental concepts to have a proper mental model, to simulate what DAX will do. This is very deceptive, because it looks like Excel formulas on steroids, but conceptually it’s very different.
Here is the problem with DAX, in a nutshell:
This example below is a perfect example of that sharp rise in learning curve, and dealing with foreign concepts like calculated columns, measures, applied filters, and evaluation contexts.
So, one of the things I’m hoping to catalog are example where DAX is a giant pain if you don’t know what you are doing. People make it look really simple and smooth, and that can be frustrating sometimes. Let’s see more failures!
How do I GROUPBY in DAX?
I need to sum an amount column, grouped by a column
Measure 1 :=
“Total AR Amt Paid calc”, SUM ( det[amt] )
I’m getting a syntax error
So automatically, something seemed off to me. Measures are designed to return a single value, given the filter context that’s applied to them. That means you almost always need some aggregate function at an outer level. But based on the name, you wouldn’t necessarily expect GROUPBY to return a single value. It would return values for each grouping instance.
If we take a look at the definition for GROUPBY(), we see it returns a table, which makes sense. But if you are new to DAX, this is really unintuitive because DAX works primarily in columns and tables. This is a really hard mental shift, coming from SQL or Excel.
What do you really want?
None of this made any sense to me. Why would you try to put a GROUPBY in a measure? That’s like trying to return an entire table for a KPI on a dashboard. It just doesn’t make sense. So I asked John what he was trying to do.
He wanted to add another column on the right, that summed up all of the amounts for transactions with the same document. In SQL, you’d probably do this using a Window function with a SUM aggregate, like here.
Calculated columns versus measures
This highlights another piece of DAX that is unintuitive. You have two ways of adding business logic: calculated columns and measures. The both use DAX, both look similar and are added in slightly different spots.
But semantically and technically, they are very different beasts. Calculated columns are ways of extending the table with new columns. They are very similar to persisted, computed columns in SQL. And they don’t care at all about your filters or front-end, because the data is defined at time of creation or time of refresh. Everything in a calculated column is determined long before you are interacting with them.
Measures on the other hand, are very different. They are kind of like custom aggregate functions, like if you could define your own version of SUM. But to carry the analogy, it would be like if you had a version of SUM that could manipulate the filters you applied in your WHERE clause. It gets weird.
My point is, if you don’t grok the difference between calculated columns and measures, you will never be able to work your way around the problem. You will be forced to grope and stumble, like someone crawling in the dark.
Filter context versus row context
So in this case we’ve determined we actually want to extend the table with a column, not create a free-floating measure. Now we run headlong into our next conceptual problem: evaluation contexts.
In DAX there are two types of evaluation contexts: row contexts and filter contexts. I won’t go too deep here, but they define what a formulas can “see” at any given time, and in DAX there are many ways to manipulate these contexts. This is how a lot of the time intelligence stuff works in DAX.
In this case, because we are dealing with a calculated column, we have only a row context, not filter context. Essentially, the formula can only see stuff in the same row. Additionally, if we use an aggregate like SUM, it only cares about the filter context. But the filter context comes from user interaction. Because this data is defined way before that, there is no filter context.
This is another area, where if you don’t understand these concepts you are SOL. Again, for the newbie, DAX is a pain.
What’s the solution?
So what is the ultimate solution to his problem? There are probably better ways to do it, but here is a simple solution I figured out.
SUM ( Source_data[Amount] ),
ALL ( Source_data ),
Source_data[Document] = EARLIER ( Source_data[Document] )
The first manipulation is to run ALL against the table, to undo any filters applied to it. In this case, the only filter is our converted row context. (confused yet?)
The next manipulation is to use EARLIER (which is horribly named) to get the value from the earlier row context. In this case we are filtering ALL the rows, to all of them that have the same document. Then, finally we apply the SUM, which “sees” the newly filtered rows.
Here is what we get as a result:
How do we verify that?
A fourth pain with DAX is that it’s very hard to look at intermediate stages of a process, like you can with SQL or Excel formulas, but in this case we have a way. If we convert our SUM to a CONCATENATEX, we can output all the inputs as a comma separated list. This gives us a slightly better idea of what’s going on.
What’s the point?
My point is, that DAX, despite it’s conciseness and richness is hard to start using. Even basic tasks can require complex concepts, and that was a big frustration point for me. You can’t just google GROUPBY and understand what’s going on.
Again, check out my presentation I did for the PASS BI virtual group. I tried to cover all the annoying parts that people new to DAX will run into. That and buy a book! you’ll need it.
So, I said to Brent, “How many people attended, I want to update my speaking log.” He said, “210? I’ll get you the data tomorrow.”
Here’s what he gave me this:
Power Query to the rescue
Normally this would be a giant pain to work with. When it comes to data quality, this is quite the tohubohu. Thankfully, I can clean things up quickly with Power Query.
So, first I’m going to click on the data and select Add to Data Model, under the Power Pivot tab.
Excel is going to make some assumptions about what is part of the table. This is convenient for our needs, but we’ll have to find a work around when we want to scale to multiple excel files.
We can’t tell it we have headers, because it’s going to think that first row is a header. We’ll deal with that later. Once we click OK, we are taken to the Power Query / Power Pivot window.
I made a mistake
Hmm, so it looks like I made a mistake. I hope my honesty won’t lose me any izzat, or ability to command respect. I think it’s important to see how people really learn and really solve problems. So, I’m including my screw ups in this post.
Apparently, I created a linked table and I can’t see how to edit the the Power Query portion for that. A linked table is a nice way to pull raw data from the Excel workbook. It’s great for reference tables, but doesn’t solve our problem.
Let’s take a different approach. I’m going to open a blank excel workbook and pull the data into there. Okay, so let’s go to manage under the Power Pivot tab.
Next, we are going to click “Get External Data From Other Sources”
Then I’m going to scroll to the bottom and select Excel File.
Once selected, I only have the whole first sheet as an option. If I had table objects or named ranges, that would be different.
Hmmm, I still can’t find a way to edit the Power Query. Fiddlesticks!
Normally, in Power BI it would be right here:
Trying to do this in Excel is quite the boyg, or vague obstacle.
Third time is a charm
Sigh, okay let’s try this a third time. I’m going to do to the Data tab and the “Get and Transform Query”. “Get and Transform” is the new name for Power Query.
Okay, let’s try opening that Excel file. Ah, much better. Now I want to click Edit at the bottom right.
Cleaning the Data
So, First thing we need to do is get rid of all of the non-header rows at the top.
To do that, I just select Remove Rows –> Remove Top Rows.
Then I specify I want to get rid of the top 7 rows.
Next, I want to turn the actual header row into a header.
Okay, so now it looks like a real table.
Comma Delimited BS
Okay, so now we need to parse out the times someone was watching. The problem is that some people were in and out. Their entries are comma delimited. Ugh.
Okay, let’s split them up. I’m going to select Split Column –> By Delimiter
Unfortunately, splitting by column a) splits into more columns and b) you have to specify how many.
Thankfully, we can select those new columns and unpivot them.
Perfect. Now we have a row for every time a person as watching.
Okay, so now we just need to parse out the dates. First, we are going to split on the dash, and then the parenthesis.
This is starting to look good.
Now we just need to get rid of the timezone and convert it to a datetime. First we need to select Replace Values.
Lastly, we select the data type we want.
Now that are data is cleaned up, we’ll join to sessions table and do some simple data modeling. But that’s for the next blog post.
I often ask people with more experience than me “How do you stay relevant in our field?”. I joke that I live in perennial fear of being replaced someday, by a 22 year-old with no family commitments. Really, it’s a joke. Really, it’s a fear.
So, let’s talk about how to keep up with technology.
First, we must grieve.
So here’s the secret to keeping up with technology. You can’t.
There are too many technologies, too many features, too many updates:
- SQL Server 2017 is coming out this year.
- Big data technologies are so numerous as to be indistinguishable from Pokemon.
- PowerBI ships features on a weekly basis.
- Worst of all, you’ve got NoSQL which is literally the mathematical dual of SQL. It’s everything that SQL isn’t, by definition!
It’s all too much.
This realization is likely to be expressed in the 5 stages of grief.
- Denial. Other people can keep up, why can’t I? I’m learning tons of stuff all the time! This is easy.
- Anger. Why they heck are they making releases every month!? When did Microsoft go agile? I thought SQL Server versions every 2 years was bad enough.
- Bargaining. Okay, maybe if I stick to core SQL stuff, I’ll be fine. Hadoop seems like a fad. And Azure is never going to be popular with my company.
- Depression. This is impossible. I’m going to lose my job when I’m 40 years old and arthritis starts kicking in.
- Acceptance. There’s more happening than I can ever learn, but Hacker News doesn’t define my success as an IT professional.
This post isn’t about doing the impossible. It’s about making the most of your resources. That’s something that you can do.
Are you asking the right questions?
I would like to propose that it’s not about keep up with technology at all. This is a second-order goal, a proxy of sorts. Really, there are two questions at the heart of things:
- How do I keep my job?
- How do I keep my friends?
This is why technology causes so much angst. We want to learn enough that we can put food on the table; and we want to do it in short enough time that it doesn’t destroy our personal lives.
Next, we must think.
In order to get ahold of the the problem, let’s use some analogies: investments and radioactive decay.
How learning is like investing
Do you have a retirement account? If so, do you invest primarily in stocks or bonds? Why?
If you are at all young, the you invest primarily in stocks. That’s because stocks have a higher rate of growth than bond. You are trying to outrun inflation, where the value of your money is steadily decreasing. This is like your current knowledge becoming outdated. All that vb6 coding knowledge is like a pile of cash in your mattress. When I was a kid, $20 was a lot of money.
The next question is, do you invest in just one stock, like Apple? No, you diversify your portfolio. High growth stocks go up, on average. Some however, tank. High growth means high volatility. A good example is Apache Flex. It used to be a really promising application platform, until Steve Jobs killed Flash.
So we’ve got two risks to our learning portfolio: Losing value in our existing knowledge, and making the wrong choices for our new knowledge. These different risks have different mitigation strategies.
Specialization and generalization
These comparisons to bonds and stocks relates to the challenges of specialization versus generalization. We specializes to pay the bills. We generalize to keep our jobs.
Specialization is how we get paid. The reason anyone pays us is because we have skills or knowledge that is not quickly acquired. The reason consultants like David Klee make gobs of money is that they have taken a subject, like virtualizing SQL, and have gotten really good at it. To get paid more than minimum wage you are going to need some level of specialization.
Generalization is how we get paid in 10 years. You need to be specialized to get paid right now, it’s a short term investment. However, that investment decays, just like the value of money in your mattress. To get paid a decade from now, you need to broaden your horizons. If you are a data person, it might mean learning R and python. It might mean learning Azure. Heck, it might mean PowerShell and Docker.
The tension here is that you need both. You need paid now AND in 10 years. Kevin Feasel talks about trying to find that right balance. I’m not here to tell you what that balance is. What’s important is that each has different constraints. Specialization requires focus. Generalization requires time. More on that later.
How learning is like radioactive decay
Let’s take another approach.
Our knowledge has a limited shelf-life. Allen White said, in the SQL Data Partners podcast, that you have to retool yourself every 5 years. In college, I remember joking that half of what you knew would be useless in 5 years.
Well, what if we took that literally? How would we model that? How would we think about that?
In nuclear physics, there is the idea of a half-life. You have radioactive material that decays in half every X units of time. Why not apply this concept to IT? The half-life in this example then would be 5 years.
So the next question is this: if half of what you learn is either irrelevant or forgotten in 5 years, how much do you need to learn in a given year, to keep steady? Let’s ask our friends at Wolphram Alpha.
If x is our rate of decay and our half-life is 5 years, we can work backwards from that and solve for x. In this case, x equals 87%.
So, what that means is that if today you know 100 relevant things, then a year from now you’ll only know 87 relevant things. That’s like going from a solid A+ to a weak B+. You just lost a whole grade! Not good.
Below is a curve showing what this model looks like over 10 years.
Changing the math
Well, this isn’t great. What if I want to know 120 things? How can we do that? One option is to learn more things.
Learn more things
If normally we have to learn 13 things each year, then we have to learn another 20 things on top of that. Or, if you are patient, you can learn an extra 4 things each year, and eventually you will get there.
Slow the decay
Another option is to change the rate of decay. Instead of learning more things each year, what if we didn’t have as much decay? If we can slow that rate of decay just a little bit, to a half-life a 6 years instead of 5 years, we’ll have the same effect. That means that instead of brute-forcing things, we might be able to be smarter.
What does this all mean?
So that gives a path forward. We need to either
- Increase the number of relevant things you can learn each year
- Or, decrease the rate of decay for relevant knowledge
These are the only three knobs we have. Either learn more stuff, learn the right stuff, or learn stuff that lasts longer. Let’s investigate all three.
How do we fix this?
Learning more things
So one solution to our dilemma is to brute force it. Let’s just learn more things and hope that they are the right ones. To do that, we can look at our inputs and constraints.
What are the constraints that affect our learning? There are 3 big ones I can think of:
If we can increase any one of these, then we might be able to increase how much we can learn in a week.
So, lets say we decide to go the simple route and go for volume. You’ve got a 168 hours in a week. You can’t make any more hours, and if you try to use all of them in a week, you are going to need some amphetamines.
So, if we can’t create more hours, how can we make more time? One option would be to cut out other activities. If you are willing to quit watching TV or playing video games, that frees up more time for learning. You could use a service like toggl.com to do a time audit and see where all of your time goes.
There is a limit to going down that path, however. You need to sleep. You need to have a social life. You need to have some fun. Like we said, you want to keep your job and keep your friends.
Another option is to multi-task. There is a lot of learning you can do that while doing other things. Things like podcasts are more about exposure than mastery. You can still get value out of them, even while mildly distracted. There are a number of times you could listen to a podcast:
- Washing dishes
This is a way to take back time you are spending on other things without giving up all your free time.
5 minute learning
Another way of reusing your time is taking advantage of those weird breaks in time. Those breaks that are too small to get anything meaningful done. The 15 minutes at the doctor’s office. The 5 minutes waiting for a meeting.
Feed readers are a great way to take advantage of these weird units of time. With Feedly, I’m able to to read a blog article during the 5 minutes I’m waiting. This is way more useful than playing candy crush or reading twitter.
Okay, so let’s say you’ve managed to find more time in the day. Unfortunately, not all learning takes the same amount of energy. Reading Twitter is mindless. Configuring a homelab requires focus. Listening to a podcast is mindless. Making a presentation takes focus.
One way of getting more focus is to prioritize the harder learning for when you naturally have focus. For some people this is early morning. For many people this is the weekend. I know that after a long day of work, I’m wiped out.
Part of that is learning to take care of yourself. Eat healthy. Exercise. Get sleep. Focus is so easy to destroy by poor lifestyle.
Finally, if you schedule time consistently and push everything out, you can have more focus. I also find having a separate office/learning space helps with this too. Cut out the distractions. Install StayFocusd for Chrome to block timewasters.
If you are anything like me, money is your most plentiful resource of all 3. When I was kid, I had no money at all, but tons of time. When I was in college, I had pretty much no money, and a good bit less time. Now that I have been working for a while, that equation has flipped. I’ve got plenty of money, but no time to use it.
If you are an average DBA, you make plenty of money. According to the Bureau of Labor Statistics, the median salary for DBA’s is $85,000. The median for all occupations is $37,000. Think about that, that’s like 2.5x as much.
Now you may say “Hey, I don’t live in San Francisco.” or “Hey, I just got started.” Let’s take those factors into account. The market here in Pittsburgh is terrible, salary-wise. Even if you are just getting started in the field, you are probably making at least 40K.
Let’s do a little math on this. If you make 40K per year, you make roughly $20 per hour. that means your time is worth $20 per hour. If you can spend $5-10 to save an hour of your time, do it.
If a $50 book saves your 5 hours of Googling, buy it. If a Pluralsight subscription saves you 30 hours per year of frustration, buy the subscription. Don’t be afraid to spend money to save yourself time and energy.
Consider budgeting your money
If you don’t feel like you have a lot of money, I would deeply urge you to start budgeting. I used to look at my checking account to see if I had money. Then if there was money there, I would spend it. This would work until my car insurance bill would come in, and suddenly I needed $600. And just as suddenly, I didn’t have nearly as much money as I thought
Get a budgeting program. There’s some free one’s out there, but I use You Need a Budget.
Learning the right things
So we covered how to learn more things. An alternative is to learn the right things. If you learn the right thing, you’ve have less wasted effort.
Lean on curation
Sturgeon’s law says that 90% of anything is crap. That’s certainly true of learning materials. Not only is there a lot of bad training out there, but there’s a lot of content in general that just isn’t relevant. It’s cool that someone wrote an OS in Rust, but it’s probably not relevant to your job in any significant way.
If you are time poor or focus poor, you need a gatekeeper so the flood of possibilities doesn’t overwhelm you. This means depending on curation.
That curator could be you, first off. One of the reason I suggest a feed reader over Twitter is that you are able to heavily curate the content that hits your eyes. You are the one picking the blogs to read instead of the pachinko ball machine of social media and fate. You are the one deciding what’s relevant to you.
Another options is to depend on professionals in the field that you trust. Find people to follow that consistently have good material or recommendations.
The third option is to pay money. Sure, there’s some paid crap out there. But general it does act as a quality filter. Things like written books and video libraries are far less likely to be total crap, because those things have editors and investors and such. Someone was being paid to make sure it was worth making. Additionally, the author worked hard to condense the knowledge into a concise format. Be willing to pay money.
Having a plan
Okay, so now you are investing your time and energy into quality materials, but are you learning the right things? Are you learning the things that are right for you? You need to have a career plan. You need 1 and 3 year goals. Otherwise your career plan will be a spread out mess.
Figure out where you want to focus, figure out where you want to go. Pick a specialization. More of what you learn will be relevant if you know what that is.
Learning things that last longer
Okay, so we are learning more things and more of the right things. But we still have the problem that our knowledge is decaying whether from atrophy or irrelevance. This is the last step in the funnel and the last thing we can optimize.
One way to keep ahold of knowledge for longer is to lean on active learning. If something goes in through your eyes and ears and back out your fingers and mouth, you will retain it for longer.
If you want to specialize and go deeper, you need to do things like blogging, presenting,coding, etc. Reading isn’t enough. Writing isn’t enough. if you truly want to learn, you have to do.
Reddit and hacker news are good for exposure to a topic, but exposure fades quickly. Use exposure to get a lay of the land, then go deep once you find the right things to learn. If you focus on mastery, you are going to learn things in a way that last longer.
Avoiding planned obsolescence
Another issue is that certain areas of knowledge just have a faster rate of decay. Some of these are facts tied to a specific version of a technology. Another thing are unproven technologies that might turn out to just be fads. Big data is still in this phase, where there are so many technologies that will be gone forever in 5 years.
People change far more slowly than technologies, so skills relating to them last far longer. Soft skills never go out of style. Communications skills never go out of style. If you learn how to write an abstract today, that knowledge will still be relevant 40 years from now.
The fundamentals last longer too. The core basics of computer science are far more establish than this weeks JS framework. If you have solid underpinnings, you can take your knowledge of C++ to Go, for example. If you understand how a ACID and CAP theorem, The new consistency models of Cosmos DB will make sense. Under understanding of one layer beneath will allow you to jump to new technologies far quicker.
Putting it all together
To summarize what we talked about:
- Take advantage of multitasking to get back more time. Listen to podcasts while you drive or exercise.
- Invest in a pod catcher. I use itunes and an iPod nano.
- Invest in a feed reader. I use Feedly.
- Take care of yourself, physically. Diet, sleep and exercise will all improve focus.
- Schedule time to maintain focus. Treat focus as your most precious resource. Don’t browse twitter when you are firing on all cylinders.
- Be willing to spend money on curation. There is a good chance that you have more money than time or focus. Be willing to spend money to avoid wasting time.
- Understand exposure versus mastery. Spend your time on exposure and your focus on mastery.
- Pick a specialization. Have a plan on what technologies you need to go deep on. You won’t get there by accident.
- Learn things that last. Learn the fundamentals. Learn internals. Learn people skills. Do home labs.
Today I’m going to be presenting on DAX for the PASS BI Virtual Group. The focus is on all the hard mental concepts of DAX. If I could sum up the talk in one picture, it would be this:
That red area is where I banged my head when learning DAX. The learning curve shoots up wildly in the middle of learning the technology, instead of a slow gentle curve. This presentation covers the middle parts that are key to understanding DAX.
Here are the slides for the presentation: Introduction-to-DAX-2017-03-30
Here is the recording:
Here is the talk by Marco Russo I mentioned in my presentation: Optimizing Multi-Billion Row Tables in Tabular
During PASS Summit, I wrote a post about the broadening data platform. I talked about the term Data Professional, and how I feel how it describes the changes going on the in SQL space. Here’s the problem: It’s a terrible guide. It’s a great description, it’s a wonderful attitude, it’s an ambitious goal; but it’s a terrible guide.
Being a data professional means being a jack of all trades. It means being a renaissance man (or woman). It’s means a career plan that looks like this:
And then you end up with Buck Woody telling you you are trying to do too much, cut it out kid.
@SQLGene Always is. But you can’t be an expert at everything. Pick something needed, do it better than anyone.
— Buck Woody (@BuckWoodyMSFT) February 21, 2017
So that’s the problem. Sometimes broadening your horizons is really a mask for being scared of commitment. Sometimes it’s a mask for being scared of an ever-changing future. You have to bet on a horse, you can’t bet on them all. Being a data “professional” is great in theory, but in practice it turns into majoring in the “universe” (see XKCD).
— Eugene M (@SQLGene) February 10, 2017
What I am saying is that if someone asks you “Where do you want to be in 3 years?”, “everywhere” is not an answer. If someone asks you “What are you going to learn this week?”, “everything” is not an answer. So yes, generalize your skill set, who knows what you’ll be doing in 5 years. But right now you need a focus, it’s the only way to become an expert at anything.
Ultimately, I think it comes down to two quotes:
If you don’t know where you are going, any road will get you there.
-George Harrison, paraphrasing Lewis Carrol
Two roads diverged in a yellow wood,And sorry I could not travel bothAnd be one traveler,
[…..]I took the one less traveled by,And that has made all the difference.-Robert Frost, The Road Not Taken
For me, I’m looking into Data Science. The problem is I’m not sure what Data science actually is! What I know for sure is it involves R and pirate jokes. We’ll cover that in next week’s blog post.
Phew! That was nerve wracking and exciting. Here are the slides from my presentation, the excel file is at the end. I’ll update the blog post with more detail when I get a chance.
EDIT: It looks like I didn’t properly attach the Excel to the PowerPoint. This has been fixed.
Here are my slides for my PUG presentation tonight.
I’m really excited to give this presentation. I think it turned out well and covers a lot of deep content for an introductory presentation. Hopefully at some point I’ll turn it into a series of posts.