Digest | MangaDex基础设施(NAT), Mobile-1st vs Desktop-1st, UNIX文本处理

>  2021年10月01日信息消化

### MangaDex Infrastructure overview

origin: [Infrastructure overview](https://mangadex.dev/mangadex-v5-infrastructure-overview/)

Infra: network, deploy, monitor, security

![img](https://raw.githubusercontent.com/Phalacrocorax/memo-image-host/master/uPic/md-infra-public.drawio--3-.png)

This is essentially a zoomed-in diagram of the left box in the network map. Access to the production VPN is also granted only to people who actively need it. The addition of a NAT gateway ensures that if somehow a public IP of ours leaked via some SSRF-like attack, we'd only need to swap a single address to affect all hosts. And to avoid misconceptions, no, we are not using AWS S3 as object storage, that would be far too costly. We do, however, use a commercial object storage service, as we still want absolute certainty for the safety of our backups.

这基本上是网络地图中左边方框的放大图。对生产用VPN的访问也只授予那些积极需要它的人。增加一个NAT网关可以确保，如果我们的一个公共IP通过一些类似SSRF的攻击而泄露，我们只需要交换一个地址就可以影响所有的主机。为了避免误解，不，我们没有使用AWS S3作为对象存储，这将是非常昂贵的。然而，我们确实使用了商业对象存储服务，因为我们仍然希望对我们的备份的安全性有绝对的把握。

Our development environment[2](https://mangadex.dev/mangadex-v5-infrastructure-overview/#anchor-2) is essentially a carbon copy of this on a smaller scale (two backends instead of three – two Elasticsearch data-bearing nodes instead of three – and smaller everything in general). It does push its metrics/logs to the same monitoring infrastructure as the production servers but is otherwise completely unrelated (different providers, networks, databases...). While it is somewhat expensive to keep it similar enough to production, it grants us confidence to make large configuration changes when necessary. The only missing  bit would be the ability to replicate production traffic, as some bugs only happen under very high traffic by a large number of concurrent users. This is however at best difficult or nearly impossible to do.

We continuously monitor the accessibility and responsiveness of the site **across multiple major cities in the world**, and taking Europe as a baseline we observe from three times the latency in North America to almost ten times the latency in Asia, with some pronounced spikes or straight up unreachability.

We have plans to remediate this, but that will not be trivial to achieve. The idea would be to have region-local read-only mirrors, and wire it all up at an inter-region level using one of [Consul](https://www.consul.io/) or [Istio](https://istio.io/) to make secure, discrete service meshes across the globe and still prevent any lateral movement capacity to a potential intruder. Something like Varnish + Backend + MySQL/ES read replicas would go a long way towards speeding things up for our distant users.

我们有计划补救这个问题，但实现起来并非易事。我们的想法是建立区域性的只读镜像，并在区域间使用Consul或Istio进行连接，在全球范围内建立安全、独立的服务网，并防止任何潜在入侵者的横向移动能力。像Varnish+Backend+MySQL/ES读取复制这样的东西将大大加快我们的远方用户的速度。

It isn't clear yet at this point which of these would be most fit for purpose however. Since we use a couple of HashiCorp products already ([Terraform](https://www.terraform.io/) for our development environment provisioning and [Vault](https://www.hashicorp.com/products/vault) for secrets management in general), it would be nice if Consul did the trick and we could leverage their integrations.

As for logs, we ship them using [Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) to a centralised [Loki](https://grafana.com/oss/loki/) instance.

We then rely on [Ceph](https://ceph.io/en/)'s [Object Gateway](https://docs.ceph.com/en/latest/radosgw/index.html) (sometimes called Rados Gateway) for persistent storage of this data. The template for how that all ties up together looks something like this:

![img](https://raw.githubusercontent.com/Phalacrocorax/memo-image-host/master/uPic/md-monitoring.drawio.svg)

### The State Of Mobile First and Desktop First

origin [The State Of Mobile First and Desktop First](https://ishadeed.com/article/the-state-of-mobile-first-and-desktop-first/?ref=devawesome.io)

Have you often thought about whether to implement a mobile-first or a desktop-first approach? Recently, I [shared a Twitter poll](https://twitter.com/shadeed9/status/1391124105877987335) about how many uses mobile-first vs desktop-first, and the results were interesting.

The total number of votes is 648, and here are the stats:

- Mobile-first: 33.3%
- Desktop first: 21.9%
- Mix of both: 24.7%

Consider the following example.

![img](https://raw.githubusercontent.com/Phalacrocorax/memo-image-host/master/uPic/intro-1.png)

```css
/* Mobile First ------------- */
.section {
    padding: 2rem 1rem;
}

@media (min-width: 62.5rem) {
    .section {
        display: flex;
        align-items: center;
        gap: 1rem;
        padding: 4rem 2rem;
    }
}

/* Desktop First ------------- */
.section {
    display: flex;
    align-items: center;
    gap: 1rem;
    padding: 4rem 2rem;
}

@media (max-width: 62.5rem) {
    .section {
        display: block;
        padding: 2rem 1rem;
    }
}
```

##### Nav Example for mobile first and desktop first

![img](https://ishadeed.com/assets/mobile-desktop-first/mobile-vs-desktop.png)

##### Scoping Styles: A Better Approach

Yes, there is. For me, I prefer to not stick to a specific approach. Instead, I like to mix them both. That means, we need to write the base styles first and then we start thinking about what will happen for mobile and desktop. I like how Elad Shechter is calling it basics first in [this great](https://elad.medium.com/the-new-responsive-design-evolution-2bfb9b504a4e) article.

```css
.header {
    /* Base styles */
}

/* Desktop styles */
@media (min-width: 1000px) {
    .nav__toggle,
    .nav__close {
        display: none;
    }
}

/* Mobile styles */
@media (max-width: 999px) {
    .nav {
        position: fixed;
        left: 0;
        top: 0;
        width: 100%;
        height: 100%;
        background-color: #4777dB;
    }
}
```

##### CSS Grid Minmax()

Thanks to CSS grid, we can have a responsive grid layout that doesn’t depend on CSS media queries. Consider the following example:

```css
.wrapper {
  display: grid;
  grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
  grid-gap: 1rem;
}
```

This is a responsive grid that gives each item a minimum width of `200px`. Without CSS grid, we will need to use media queries to alter the width based on the viewport size.

### Effective Go

>  [HN](https://news.ycombinator.com/item?id=28735620) ｜ I love testing in Go because ss
>
>  I avoid meaningless mocking. Test structs that match interface signatures are meaningfully used to validate any state handling and/or error generation, and ensures error paths are properly exercised. In unit tests, we validate logs and metrics when appropriate in addition to return values. 
>
>  However, if you are mocking db, http or other net packages, you are likely doing it wrong. You want to know you handle an error from the db, you don't have to mock a db: you have a struct whose interface has 'GetUser(username) (*User, error)', and your test version returns an error when needed. The fact the real code will use a db is an implementation detail. **You should be able to refactor the real implementation to change from a db call to an http api call and still have valid, useful tests**. Elsewise, your unit tests are too coupled and hinder refactoring. Anyway, I love testing in Go; it is one of my favorite parts of working with it.

### UNIXテキスト処理の極意

Software Developer 2021.8

- `wc <filename>` 行数、単号数、文字数を数える
- `head -3 <filename>` 最初から３行
- `tail -3 <filename>` 最後から３行
- `grep -v '^$' <filename>`, `sed '/^$/d' <filename> `, `awk 'NF!=0' <filename>` 空行を除去
- `awk 'length>10' <filename>` １行が11文字以上ある行を出力
- `tail -r <filename>` 逆順に１行ずつ出力
- `rev -r <filename>` 行の左右反転

##### 回文の判定＋処理時間をtimeコマンドで測定する。

```bash
# readでの処理（性能的に悪い）
time while read line; do if [ "${line}" = "$(echo ${$line} | rev)" ]; then echo $line;fi;done < examination.txt

# commコマンドとbashのプロセス置換で回文を抽出する
# comm -12: 共通の部分だけ表示させたい
# 問題点: sort済みのデータを期待しているcommのために、対でないデータ同士を比較することになって
comm -12 <(sort examination.txt) <(rev examination.txt | sort)

# pasteとAWKを利用
paste examination.txt <(rev examination.txt) | awk '$1==$2{print $1}'
```

### Misc

- CSS **Custom Properties** for Cascading Variables Module

- ```css
    :root {
      --main-color: #06c;
      --accent-color: #006;
    }
    /* The rest of the CSS file */
    #foo h1 {
      color: var(--main-color);
    }
    ```

- [*The new CSS* *property*](https://www.smashingmagazine.com/2021/09/simplifying-form-styles-accent-color/?)

- `accent-color` 
  - `aspect-ratio`

- I went on a scrum course and the takeaway was basically that **feedback is a big deal,** and you should try to get some repeatedly and quickly. It's also common sense that you should have tasks written down somewhere, and that some of them are more important than others.

- [WSL | tiktok vs instagram!](https://cbinsights.us1.list-manage.com/track/click?u=0c60818e26ecdbe423a10ad2f&id=2c4d73dbf7&e=f156187b75)

- [An Overview of Dashboarding Technology in the Age of Big Data](https://towardsdatascience.com/data-visualization-in-2021-an-overview-of-dashboarding-technology-in-the-age-of-big-data-79d490beffcf)

- ![img](https://raw.githubusercontent.com/Phalacrocorax/memo-image-host/master/uPic/1*aNZAo8ZxNk7hm09bEqjUoA.png)