并发 (concurrency) 和并行 (parallelism) 是两个概念. 这两个术语的使用还有争议, 下面依照书上 (Fluent Python, 2nd Edition) 的说法, 并行是并发的子集.

A modern laptop with 4 CPU cores is routinely running more than 200 processes at any given time under normal, casual use. To execute 200 tasks in parallel, you’d need 200 cores. So, in practice, most computing is concurrent and not parallel.

首先是这篇文章

• Horace He. (2022). Making Deep Learning Go Brrrr From First Principles

内容不再赘述, 下面是纲要和一些对原文的注解. 虽然文章用 Torch 和 GPU 举例, 但基本原理依然普适.

简单介绍了模型耗时可以分为三部分

• Compute: Time spent on your GPU computing actual floating point operations (FLOPS)
• Bandwith: Time spent on moving the data from CPU to GPU, from one node to another, or even from CUDA global memory to CUDA shared memory
• Overhead: Everything else

Makefile

直接看原文

• Aniket Bhattacharyea. (2021). Makefile Tutorials and Examples to Build From
• Aniket Bhattacharyea. (2021). Creating a Python Makefile

用 setuptools

主要参考 java4python “Learn Java: Programiz” app 上的小课. Just a quick review.

主要参考 cpp4python 和 “Learn C++: Programiz” app 上的小课, 内容是简化版中的简化版.

备用

• 终极 C++避坑指南
• cppreference.com

C++ 文件的扩展名常见的有 .cpp, .cc 等.

主要参考

• Shotts, W. (2019). The Linux command line

学习目标: 能写简单脚本, 能看懂长一些的脚本. Shell 脚本不是开发语言, 难以 debug, 不适合写太长.

If you are writing a script that is more than 100 lines long, or that uses non-straightforward control flow logic, you should rewrite it in a more structured language now. Bear in mind that scripts grow. Rewrite your script early to avoid a more time-consuming rewrite at a later date. (Google Shell Style Guide)

装饰器基础, 以及常用装饰器略了, 比如 staticmethod, classmethod, property, functools.lru_cache.

随手找的一些模型部署简单例子.

• Flask+Gunicorn+Service Streamer
• Flask+Gunicorn
• Flask+gevent
• Tornado

NVIDIA 的 Triton Inference Server.

把模型文件 (支持主流格式) 放在 model repo, 由 inference server 调度, 根据模型配置的推理引擎 (ONNX, TensorRT 等) 输出结果.

下面以在 CPU 上部署 ONNX 模型为例.