You have been given a business problem and a lot of raw data. You want to engineer this data and choose the right metrics to solve this problem. Alexey: [laughs] Okay, I think that's all we have time for. So maybe last one – How can people find you? ( 1:00:03) Valerii: Level five. But there is no clear way, nobody will tell you “You’re level five. You'll be trained for level five.” Of course, there’s always some margin. So you might end up being level four, but still go through this interview, because you were on the brink between four and five. ( 55:13) Valerii: Yeah, true. Good catch. Yes, level five is a Senior in terms of the level on Facebook, which means that, if you're on this level, it is an honorary thing to be on this level forever. So if you ended on level four, it was probably because of the ML system design interview. This interview tells the interviewer (Facebook or Google, or whatever company) your ability to have an overview of the system. In 45 minutes, you have to be able to tell a story – almost a monolog of yours – about how you will build the system and touch very different points. ( 11:23)

You can read the web-friendly version of the book here. You can find the source code on GitHub. The Discord to discuss the answers to the questions in the book is here. Alexey: Yeah, and then on the ML system design, you would talk through the log loss and things like this. ( 24:21) Valerii: I was. I don't know. I am trying to take a look now because there is an exponential decay if you don't compete, and what is even more important, if you don't win, your score is decaying. Kaggle is an addiction. So the best way is not to go there because you can suddenly find yourself doing the Kaggle again. ( 2:32) I have been using your github repo to prep for my interviews and got an offer with NVIDIA with their data science team. Thanks again for your help!This course aims to provide an iterative framework for developing real-world machine learning systems Valerii: Yes. To approximate, “Can you move directly to your goal? Or can you approximate moving to your goal?” Also, the thing is that – if a metric becomes your goal, with some time, it usually ceases to be a good metric. ( 43:17) Product leadership: do you design ML solutions to provide for the needs of users and product designers

Valerii: The best way is not even to ask, but to say “My assumption is that. Do you agree with that or not?” You see, you asked the question, but actually, you’ve made an assumption. You say “Are you okay with that?” Because you've been given some information. Of course, in the real world, we would gather the context because context can make everything very different. Because imagine, like in the case of fraud – if you receive a label within minutes, it's very different to receiving a label within months. It affects everything. But you could make an assumption, you say, “My assumption is that.” To build, you might be making many assumptions and nobody prevents you from making assumptions, which will make your life easier. ( 21:10) Each ML use case in your organization has been deployed using its own workflow, and you want to lay down the foundation (e.g., model store, feature store, monitoring tools) that can be shared and reused across use cases.Valerii: Meanwhile, on the machine learning design interview, usually, the thing is to understand how you would build it from the machine learning perspective. Let's give an example. Let's say that one of the questions is “How would you build a model that has to catch fraud on the platform?” Let's imagine the best way. If I had a crystal ball that tells me with 100% accuracy if a transaction is fraudulent or not, then the problem is solved, right? I just take the ball, I run the transaction through the ball – the ball tells me one or zero. So that's done. However, we understand that will never happen. There will always be some discrepancy. ( 13:58)

Alexey: [laughs] I might be wrong with using these words. I think the recruiter probably used different words. But the reason for me failing the process – the whole interview – was machine learning system design. Not the others. I was afraid about the others. But in the others, I did well, but I failed that one. And the reason there was because the interviewer expected me to talk about actual machine learning. Instead, we talked about metrics, heuristics, and then I didn't have enough time to actually cover machine learning. Yeah, so what do you think about this? Is this typical for the process? Is it expected? ( 28:28) Let A and B be events on the same sample space, with P (A) = 0.6 and P (B) = 0.7. Can these two events be disjoint?

FAQ

This repo is written based on REAL interview questions from big companies and the study materials are based on legit experts i.e Andrew Ng, Yoshua Bengio etc. Alexey: Then I suggested some heuristics. I don't remember, maybe suggesting clustering people by interests and then selecting the most popular points of interest for each cluster, specifically, and then recommending this to the user. Then I suggested some other heuristics on top of that. At the end, I had a bit of time to talk about actual machine learning. At the time, I thought I really nailed it. ( 26:25)

If you’re discussing a recommendation system, the first stage is looking at a large set of items to recommend and narrowing this down. If the universe of possible items to recommend is very large, then it’s not feasible to evaluate each one in real time. You need a heuristic to generate an initial list of candidates. What are some common heuristics? Valerii: That's true. But also not just that. The thing is, it's one of the most important interviews. Let's say that you can fail the cold interview – to some extent, since you can fail on different scales – and still, they can push you further. So, it's a critical step. ( 13:18) System design VS ML System design Valerii: I think algorithms are just one of the smallest parts, 1-5%. Well, I was speaking with a candidate recently and I told him “Look, imagine that you're a machine learning engineer in the company for two years,” He said, “Okay, okay. I can imagine that.” “Imagine that you spend an immense amount of time creating an algorithm – finding the best algorithm, setting up the loss function, the metrics, all the rest. It took you a humongous amount of time – two weeks. And you’re in the company for two years. What do you do?” Right? ( 48:07) Machine learning systems design is the process of defining the software architecture, infrastructure, Good understanding of machine learning algorithms (e.g. at least one of CS229, CS230, CS231N, CS224N

What is the ML interview?

Valerii: I don't get the question. But by itself, it's a system. Every machine learning model, it's not like a model – it's a whole system, because you have features coming to the model, the model outputs something, these outputs also have to be taken into account. There might be A/B testing here, and I did feature preparation here. So it's a whole system. ( 25:42) Some companies may not care at all about infrastructure for this interview, while others may actually combine ML with Distributed Systems. Make sure you’re clear on expectations for how much you should discuss the actual infrastructure for the interview. Even if infrastructure isn’t important, you should still keep in mind the limitations that modern computing imposes. No, you won’t be able to run a million high dimensional pictures through a Resnet model in real time. See the Infrastructure Components section below for some important ML infrastructure. Model Development Logistic regression: Fast to train, very compact, but only finds linear relationships between features.