> 2021年06月30日信息消化 ### 如何直观地解释 backpropagation 算法？ 知乎 https://www.zhihu.com/question/27239198/answer/43560763 > backprop的发展路线大概是，1974年有个Harvard博士生Paul Werbos首次提出了backprop，不过没人理他，1986年Rumelhart和Hinton一起重新发现了backprop，并且有效训练了一些浅层网络，一下子开始有了名气。那个时候的backprop从现在看来并不是个很清晰的概念，把梯度和更新一块打包了。 BackPropagation算法是多层神经网络的训练中举足轻重的算法。 简单的理解，它的确就是复合函数的链式法则，但其在实际运算中的意义比链式法则要大的多。 机器学习可以看做是数理统计的一个应用，在数理统计中一个常见的任务就是拟合，也就是给定一些样本点，用合适的曲线揭示这些样本点随着自变量的变化关系。深度学习同样也是为了这个目的，只不过此时，样本点不再限定为(x, y)点对，而可以是由向量、矩阵等等组成的广义点对(X,Y)。而此时，(X,Y)之间的关系也变得十分复杂，不太可能用一个简单函数表示。然而，人们发现可以用多层神经网络来表示这样的关系，而多层神经网络的本质就是一个多层复合的函数。借用网上找到的一幅图[1]，来直观描绘一下这种复合关系。  其对应的表达式如下：  上面式中的Wij就是相邻两层神经元之间的权值，它们就是深度学习需要学习的参数，也就相当于直线拟合y=k*x+b中的待求参数k和b。 梯度下降法需要给定一个初始点，并求出该点的梯度向量，然后以负梯度方向为搜索方向，以一定的步长进行搜索，从而确定下一个迭代点，再计算该新的梯度方向，如此重复直到cost收敛。 假设我们把cost函数表示为![[公式]](https://www.zhihu.com/equation?tex=H%28W_%7B11%7D%2C+W_%7B12%7D%2C+%5Ccdots+%2C+W_%7Bij%7D%2C+%5Ccdots%2C+W_%7Bmn%7D%29), 那么它的梯度向量[2]就等于![[公式]](https://www.zhihu.com/equation?tex=%5Cnabla+H++%3D+%5Cfrac%7B%5Cpartial+H%7D%7B%5Cpartial+W_%7B11%7D+%7D%5Cmathbf%7Be%7D_%7B11%7D+%2B+%5Ccdots+%2B+%5Cfrac%7B%5Cpartial+H%7D%7B%5Cpartial+W_%7Bmn%7D+%7D%5Cmathbf%7Be%7D_%7Bmn%7D), 其中![[公式]](https://www.zhihu.com/equation?tex=%5Cmathbf%7Be%7D_%7Bij%7D)表示正交单位向量。为此，我们需求出cost函数H对每一个权值Wij的偏导数。而**BP算法正是用来求解这种多层复合函数的所有变量的偏导数的利器**。 我们以求e=(a+b)*(b+1)的偏导[3]为例。 它的复合关系画出图可以表示如下：  在图中，引入了中间变量c,d。 为了求出a=2, b=1时，e的梯度，我们可以先利用偏导数的定义求出不同层之间相邻节点的偏导关系，如下图所示。  利用链式法则我们知道： ![[公式]](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+e%7D%7B%5Cpartial+a%7D%3D%5Cfrac%7B%5Cpartial+e%7D%7B%5Cpartial+c%7D%5Ccdot+%5Cfrac%7B%5Cpartial+c%7D%7B%5Cpartial+a%7D)以及![[公式]](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+e%7D%7B%5Cpartial+b%7D%3D%5Cfrac%7B%5Cpartial+e%7D%7B%5Cpartial+c%7D%5Ccdot+%5Cfrac%7B%5Cpartial+c%7D%7B%5Cpartial+b%7D%2B%5Cfrac%7B%5Cpartial+e%7D%7B%5Cpartial+d%7D%5Ccdot+%5Cfrac%7B%5Cpartial+d%7D%7B%5Cpartial+b%7D)。 链式法则在上图中的意义是什么呢？其实不难发现，![[公式]](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+e%7D%7B%5Cpartial+a%7D)的值等于从a到e的路径上的偏导值的乘积，而![[公式]](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+e%7D%7B%5Cpartial+b%7D)的值等于从b到e的路径1(b-c-e)上的偏导值的乘积加上路径2(b-d-e)上的偏导值的乘积。也就是说，对于上层节点p和下层节点q，要求得![[公式]](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+p%7D%7B%5Cpartial+q%7D)，需要找到从q节点到p节点的所有路径，并且对每条路径，求得该路径上的所有偏导数之乘积，然后将所有路径的 “乘积” 累加起来才能得到![[公式]](https://www.zhihu.com/equation?tex=%5Cfrac%7B%5Cpartial+p%7D%7B%5Cpartial+q%7D)的值。 大家也许已经注意到，这样做是十分冗余的，因为很多**路径被重复访问了**。比如上图中，a-c-e和b-c-e就都走了路径c-e。对于权值动则数万的深度模型中的神经网络，这样的冗余所导致的计算量是相当大的。 **同样是利用链式法则，BP算法则机智地避开了这种冗余，它对于每一个路径只访问一次就能求顶点对所有下层节点的偏导值。** 从最上层的节点e开始，初始值为1，以层为单位进行处理。对于e的下一层的所有子节点，将1乘以e到某个节点路径上的偏导值，并将结果“堆放”在该子节点中。等e所在的层按照这样传播完毕后，第二层的每一个节点都“堆放"些值，然后我们针对每个节点，把它里面所有“堆放”的值求和，就得到了顶点e对该节点的偏导。然后将这些第二层的节点各自作为起始顶点，初始值设为顶点e对它们的偏导值，以"层"为单位重复上述传播过程，即可求出顶点e对每一层节点的偏导数。 ### HOW TO WORK HARD 原文：[HOW TO WORK HARD](http://paulgraham.com/hwh.html) **One thing I know is that if you want to do great things, you'll have to work very hard**. I wasn't sure of that as a kid. Schoolwork varied in difficulty; one didn't always have to work super hard to do well. And some of the things famous adults did, they seemed to do almost effortlessly. Was there, perhaps, some way to evade hard work through sheer brilliance? Now I know the answer to that question. There isn't. Bill Gates, for example, was among the smartest people in business in his era, but he was also among the hardest working. "I never took a day off in my twenties," he said. "Not one." It was similar with Lionel Messi. He had great natural ability, but when his youth coaches talk about him, what they remember is not his talent but his dedication and his desire to win. 例如，比尔-盖茨是他那个时代商业领域最聪明的人之一，但他也是最勤奋的人之一。他说："我在二十多岁的时候从来没有休息过一天，"他说。"一次都没有。" 莱昂内尔-梅西的情况也类似。他有很强的自然能力，但当他的青年教练谈到他时，他们记住的不是他的天赋，而是他的奉献精神和求胜欲望。 P. G. Wodehouse would probably get my vote for best English writer of the 20th century, if I had to choose. Certainly no one ever made it look easier. But no one ever worked harder. At 74, he wrote > with each new book of mine I have, as I say, the feeling that this time I have picked a lemon in the garden of literature. A good thing, really, I suppose. Keeps one up on one's toes and makes one rewrite every sentence ten times. Or in many cases twenty times. Sounds a bit extreme, you think. And yet Bill Gates sounds even more extreme. Not one day off in ten years? These two had about as much natural ability as anyone could have, and yet they also worked about as hard as anyone could work. You need both. Strangely enough, the biggest obstacle to getting serious about work was probably school, which made work (what they called work) seem boring and pointless. I had to learn what real work was before I could wholeheartedly desire to do it. That took a while, because even in college a lot of the work is pointless; there are entire departments that are pointless. But as I learned the shape of real work, I found that my desire to do it slotted into it as if they'd been made for each other. 奇怪的是，认真工作的最大障碍可能是学校，它使工作（他们所谓的工作）看起来很无聊，毫无意义。我必须先了解什么是真正的工作，才能全心全意地渴望去做。这需要一段时间，因为即使在大学里，很多工作也是毫无意义的；有整个部门都是毫无意义的。但是，当我了解到真正的工作的形状时，我发现我对工作的渴望与之契合，仿佛它们是为彼此而生的。 The bigger question of what to do with your life is one of these problems with a hard core. **There are important problems at the center, which tend to be hard, and less important, easier ones at the edges**. So as well as the small, daily adjustments involved in working on a specific problem, you'll occasionally have to make big, lifetime-scale adjustments about which type of work to do. And the rule is the same: working hard means aiming toward the center — toward the most ambitious problems. By center, though, I mean the actual center, not merely the current consensus about the center. The consensus about which problems are most important is often mistaken, both in general and within specific fields. If you disagree with it, and you're right, that could represent a valuable opportunity to do something new. 不过，我所说的中心，是指实际的中心，而不仅仅是目前关于中心的共识。关于哪些问题是最重要的共识往往是错误的，无论是在一般情况下还是在特定领域内。如果你不同意这个共识，而且你是对的，这可能代表了一个做新事情的宝贵机会。 The more ambitious types of work will usually be harder, but although you should not be in denial about this, neither should you treat difficulty as an infallible guide in deciding what to do. If you discover some ambitious type of work that's a bargain in the sense of being easier for you than other people, either because of the abilities you happen to have, or because of some new way you've found to approach it, or simply because you're more excited about it, by all means work on that. Some of the best work is done by people who find an easy way to do something hard. As well as learning the shape of real work, you need to figure out which kind you're suited for. And that doesn't just mean figuring out which kind your natural abilities match the best; it doesn't mean that if you're 7 feet tall, you have to play basketball. What you're suited for depends not just on your talents but perhaps even more on your interests. A [deep interest](http://paulgraham.com/genius.html) in a topic makes people work harder than any amount of discipline can. 更有雄心的工作类型通常会更难，但尽管你不应该否认这一点，你也不应该把困难作为决定做什么的无懈可击的指南。如果你发现某些雄心勃勃的工作类型对你来说比其他人更容易，无论是因为你碰巧拥有的能力，还是因为你找到了一些新的方法，或者仅仅是因为你对它更感兴趣，都可以在这方面下功夫，这是很便宜的。一些最好的工作是由那些找到一个简单的方法来做一些困难的人完成的。 除了学习实际工作的形式之外，你还需要弄清楚你适合哪种工作。这并不仅仅意味着要弄清楚你的自然能力与哪种工作最匹配；这并不意味着如果你身高7英尺，你就必须打篮球。你适合做什么，不仅取决于你的才能，也许更取决于你的兴趣。对某一主题的浓厚兴趣使人们比任何数量的纪律都更努力工作。 The difficulty of figuring out what to work on varies enormously from one person to another. That's one of the most important things I've learned about work since I was a kid. As a kid, you get the impression that everyone has a calling, and all they have to do is figure out what it is. That's how it works in movies, and in the streamlined biographies fed to kids. Sometimes it works that way in real life. Some people figure out what to do as children and just do it, like Mozart. But others, like Newton, turn restlessly from one kind of work to another. Maybe in retrospect we can identify one as their calling — we can wish Newton spent more time on math and physics and less on alchemy and theology — but this is an [illusion](http://paulgraham.com/disc.html) induced by hindsight bias. There was no voice calling to him that he could have heard. 弄清工作内容的难度因人而异，差异巨大。这是我从小到大在工作方面学到的最重要的东西之一。作为一个孩子，你的印象是每个人都有自己的使命，他们所要做的就是弄清楚它是什么。电影里是这样的，给孩子们的精简传记里也是这样的。有时在现实生活中也是如此。有些人在小时候就想好了要做什么，然后就去做，比如莫扎特。但其他人，如牛顿，则不安分地从一种工作转向另一种。也许回过头来，我们可以确定其中一个是他们的使命--我们可以希望牛顿花更多的时间在数学和物理学上，而不是在炼金术和神学上--但这是由事后诸葛亮的偏见所引起的[幻觉](http://paulgraham.com/disc.html)。没有任何声音在呼唤他，他可以听到。 So while some people's lives converge fast, there will be others whose lives never converge. And for these people, figuring out what to work on is not so much a prelude to working hard as an ongoing part of it, like one of a set of simultaneous equations. For these people, the process I described earlier has a third component: along with measuring both how hard you're working and how well you're doing, you have to think about whether you should keep working in this field or switch to another. If you're working hard but not getting good enough results, you should switch. It sounds simple expressed that way, but in practice it's very difficult. You shouldn't give up on the first day just because you work hard and don't get anywhere. You need to give yourself time to get going. But how much time? And what should you do if work that was going well stops going well? How much time do you give yourself then? 因此，虽然有些人的生活快速收敛，但也会有一些人的生活从未收敛。对这些人来说，弄清楚要做什么工作与其说是努力工作的前奏，不如说是努力工作的持续部分，就像一组同位素方程中的一个。对这些人来说，我前面描述的过程还有第三个组成部分：在衡量你工作的努力程度和你做得如何的同时，你必须考虑你是否应该继续在这个领域工作，还是转到另一个领域。如果你努力工作，但没有得到足够好的结果，你就应该转换。这样表述听起来很简单，但实际上却很难。你不应该在第一天就放弃，只因为你努力工作却没有任何进展。你需要给自己一些时间，让自己有所作为。但是有多少时间呢？如果原本进展顺利的工作不再顺利，你又该怎么办？那时你会给自己多少时间？ What even counts as good results? That can be really hard to decide. If you're exploring an area few others have worked in, you may not even know what good results look like. History is full of examples of people who misjudged the importance of what they were working on. **The best test of whether it's worthwhile to work on something is whether you find it interesting.** That may sound like a dangerously subjective measure, but it's probably the most accurate one you're going to get. You're the one working on the stuff. Who's in a better position than you to judge whether it's important, and what's a better predictor of its importance than whether it's interesting? For this test to work, though, you have to be honest with yourself. Indeed, that's the most striking thing about the whole question of working hard: how at each point it depends on being honest with yourself. Working hard is not just a dial you turn up to 11. It's a complicated, dynamic system that has to be tuned just right at each point. You have to understand the shape of real work, see clearly what kind you're best suited for, aim as close to the true core of it as you can, accurately judge at each moment both what you're capable of and how you're doing, and put in as many hours each day as you can without harming the quality of the result. This network is too complicated to trick. But if you're consistently honest and clear-sighted, it will automatically assume an optimal shape, and you'll be productive in a way few people are. ### Four Fake Nice Gestures That Are Actually Manipulative 原文：[Four Fake Nice Gestures That Are Actually Manipulative](https://medium.com/mind-cafe/four-fake-nice-gestures-that-are-actually-manipulative-a168e1bc9bb1) #### The Ben Franklin Effect Ben Franklin had a political enemy who was making speeches against him. Franklin [remembered an](https://en.wikipedia.org/wiki/Ben_Franklin_effect#cite_note-2) old quote, “He that has once done you a kindness will be more ready to do you another than he whom you yourself have obliged.” Then he asked that political enemy to loan him a rare book. Then, Ben returned it to him with a nice thank you note. Suddenly, their political rivalry vanished and Franklin had a new friend. 本-富兰克林有一个政敌，正在发表反对他的演讲。富兰克林[记住了](https://en.wikipedia.org/wiki/Ben_Franklin_effect#cite_note-2)一句老话："曾经对你有过恩惠的人，会比你自己有恩惠的人更愿意对你有恩惠。" 然后他要求那个政敌借给他一本罕见的书。然后，本把书还给了他，并写了一封漂亮的感谢信。突然间，他们的政治竞争消失了，富兰克林有了一个新朋友。 Asking for a favor is a conveyance of trust and need. That person feels chosen. You are suggesting they have something you don’t: knowledge, ability, resources. It appeals to their insecurities. If you don’t know the requester or are angry at them, it creates a cognitive **dissonance**: why am I helping someone I don’t know or am mad at? 求人帮忙是一种信任和需求的传达。那个人感觉被选中了。你在暗示他们拥有你所没有的东西：知识、能力、资源。这吸引了他们的不安全感。如果你不认识请求者或对他们很生气，就会产生一种认知上的不协调：为什么我要帮助一个我不认识或生气的人？ Your brain watches you behaving in conflict and creates a subconscious conclusion: you must like the person. After all, we ask favors of people we trust and care for. Beware of favor requests from people who have something to gain from you. 你的大脑看着你在冲突中的行为，产生了一个潜意识的结论：你一定是喜欢这个人。毕竟，我们会向我们信任和关心的人请求恩惠。小心那些从你身上有好处的人的恩惠请求。 ##### Recap for memory: four fake nice signs of manipulative behavior 1. Asking you to do favors to endear themselves to you. Beware of the Ben Franklin Effect. 2. Bestowing you with gifts as a way to create a debt to them. Humans are reciprocal in nature. Strings are often hidden. 3. They pretend there was never a problem in the first place. They gaslight you into questioning reality. 4. They show just enough interest in you to keep your attention. They are feeding you crumbs with no intention of providing the meal. ### 一点收获 - Fat loss - 1. Don't snack; 2. Reduce stress; 3. HIIT rather than cardio. - 费曼学习法/The Feynman Technique: - 了解领域概念 - 复述概念 - 此方法并不使用所有学科。 - Github为Open Source加了一个Discussion的功能，太酷了...feature feedback/bug feedback 可以整合在github中？ - https://docs.github.com/en/discussions/quickstart - You have to understand the shape of real work, see clearly what kind you're best suited for, aim as close to the true core of it as you can, accurately judge at each moment both what you're capable of and how you're doing, and put in as many hours each day as you can without harming the quality of the result.