• Fabric project ride-along: Week 0 – let’s wing it

    |

    I’ve written before about struggling to learn Azure Synapse, and I’ve struggled as well with getting excited about Microsoft Fabric. I think the pitch and the potential of Microsoft Fabric is real. The issue is that it solves problems I don’t have. In my work, I don’t deal with data so big that Power BI can’t handle it. I don’t deal with data so unstructured that Power Query can’t handle it.

    But I know I need to learn Fabric. Power BI is a part of Fabric, the integrations are only going to continue to improve. If nothing else, I need to be able to tell customers if they should look into using Fabric or not. So what do you do when there is a technology you aren’t excited about, but have to learn?

    One solution is to get certified. In the past, I’ve written about how I find certs to be useful learning paths and something concrete to focus on. Last week they announced the DP-600 certification which looks promising for that. Another option is to take on a work project that is a bit of a stretch and then learn on the job. As a consultant, that’s always a bit of a catch-22 because you are selling yourself based on expertise you theoretically already have. The last option is to create a homelab and a side project.

    The challenge, though, is what do you put up there for a homelab? A lot of publicly available data is boring, purely descriptive, and/or already cleaned. For simple descriptive reporting, that’s perfectly fine. But for Fabric you want big data, ugly data, changing data. In comes the Magic the Gathering card game and a little data tracking project called 17lands.

    Magic the Gathering and its big data revolution

    Magic the Gathering, if you don’t know, is a competitive trading card game. With the rise of its online client, MTG Arena, it’s been going through a similar revolution like baseball and Sabermetrics (or so I assume, I’m not a sports guy). Now, instead of speculating which cards from a new set are the best, it’s possible to track in that in real-time thanks to a project called 17lands which collects data from players who opt in.

    This has allowed for fascinating analysis. Even if you don’t play, I recommend checking out this video below. It’s fascinating to see how the “metagame” of a format evolves over time as people realize which cards are good and which cards are bad. It also allows for a lot of amateur analysis, for good and for bad. Then every 4 months it happens all over again with a new release.

    This data seems ideal for a few reasons, first the raw data is big but manageable. A single “season” is 10 GB uncompressed, and 0.25 GB compressed. I did learn that Excel will try its best to open 10GB file, yell at you about too many rows, and then show you’re the first million. The 40x compression also suggests that the data is very denormalized and would benefit from some normalization.

    It did end up showing me the first million rows

    The second reason is that the schema is a mess. The data has over 600 columns, many of which are numerical flags for each individual possible card, which changes from season to season. Trying to manage this in Power Query is theoretically doable but likely very frustrating.

    Finally, it’s something I’m interested in. MTG_ds on Twitter is constantly posting graphics like this (increasing wordiness of cards each release), with insights hiding behind the high level numbers.

    A chart showing increasing wordiness of cards over time

    There are actually questions that people are interested in, that aren’t easy to answer. I like to make replayable subsets of cards called “cubes“, so being able to do things like mathematically optimizing based on cost and fun are interesting to me.

    Calling my shot

    I think with this sort of thing, it’s important to document your expectation and pain points, because you only get to be a newbie once. I’ll try to write down my expectations ahead of time so we can see where I’m wrong.

    From what I’ve seen so far, I expect the learning path at learn.microsoft.com to be very helpful in getting oriented. I expect a lot of content online to be frustrating, because so much of it assumes you have a data lake and know what you are doing.

    Speaking of which, my background is as a former DBA and now Power BI consultant. I’ve never touched ADF, data lakes, or ML in and professional capacity. As the title says, I’m going to be winging it. What I do have, however, is experience having to learn a new technology in 2-3 months (see the course below) and experience breaking down big BI projects into smaller chunks.

    The one year I needed to pay the bills and made courses on technology I had never seen before.

    I hope you enjoy watching the ride and let me know if there’s anything specific you’d like me to include.

  • Hustle culture, welcoming everyone, and taking care of yourself.

    |

    Imagine for a moment that you went to the gym, and everyone there was really fit. Muscular and tone. You look around for cameras because you think you might be at a photo shoot. How would that make you feel? You might be excited because you are in the right place to improve. Or you might be like me and worry about fitting in, worry about annoying folks, worry that your goal of being a little bit healthier is too small.

    That’s how YouTube is for me. I look at Guy in a Cube, and SQLBI, and Curbal; and I feel inferior. I think I’ll never charge those rates, I’ll never have that many subscribers, I’ll never reach that pinnacle. It’s demoralizing. It’s also utter horsecrap. I’m pretty sure they all see me as a peer.

    Imagine again that in that gym you go to talk to one of the instructors and they say “If you want to become an Olympic level athlete, you will have to train for YEARS. If you want to be the best of the best you need 10,000 hours of dedicated practice. You may have to spend 20, 40, or 60 hours per week training to reach that level!”

    Would you feel pumped up? Would you feel inspired? Would you feel excited?

    Personally, I would leave that gym. And I would never come back, because it wasn’t a place where I belonged.

    The problem with hustle culture

    Hustle culture, like most cultures, has some admirable values. Grit, determination, and self-reliance are positive virtues. But taken to an extreme, it places all on the onus on the individual to “work” their way through any problem. In the past, I’ve hurt myself and others because of this mindset.

    I worked at my last job far longer than I should have, because I thought if I just worked harder and more hours, I could fix it all. I thought I just needed to get better, faster, smarter. In reality, the kindest things I did for everyone was quit my job.

    I’m painfully German, so the way I show love is through acts of service, not kind words, not quality time. My German grandpa showed me love by having me pour concrete. I’m not sure if he ever said “I love you”. And in my marriage, I thought I was only of value if I was “doing” things. I didn’t value just being present, and that led to some bumps the first few years of our marriage. I always thought I had to be “doing” something to earn my place.

    Hustle culture places all of the responsibility on the individual. It ignores the role of community and society. It blames the individual for all of their problems. We as a community can do more than that. We can take on each other’s challenges.

    Welcoming everyone

    I hope you will forgive the religious reference that follows, but I believe that if you take the Christian faith seriously, truly seriously, then you have to believe that every single person is important. Every single person is made in the image of God and deserving of respect. Regardless of how many hours they work or their career aspirations.

    It’s good to inspire greatness, but it’s better to remind people that they are already great.

    And if you continue to take that faith seriously, then you have to be willing to meet people wherever they are, in whatever circumstances they are in, and be present. Be present and witness their suffering. See the single parent that is trying to manage parenting and a job at the same time. See the woman whom everyone assumes that she works in marketing or HR, and see her struggles and anger. See the person with depression or anxiety that struggles to get out of bed, much less make it through the work day.

    To welcome everyone, we have to see everyone. And to see everyone, we have to tolerate their pain and suffering, and bear part of it ourselves.

    Taking care of yourself

    People making proclamations about what you should do or must do, they don’t know your life circumstances. They physically can’t. You know your limits and you should respect them. And even that inner voice in my head that compares myself to people on YouTube often forgets the full picture.

    I spent last Sunday bringing my mom over to my house so we could bury her dog. It was sad, and it was human, and it was the best way I could have spent that Sunday. Better than anything work could provide.

    In 2022, I worked too much and got myself burnt out. This year, I want to work less, take better care of myself, and stop comparing myself to subscriber counts on YouTube.

Recent Posts

Categories