六月日常记录
TinyJPEG
- 灵感启发自compressorjs，不过这个库是基于
canvas实现的，而且使用到了new Image()，这两个东西都是无法在worker中使用的，虽然canvas.toBlob理论上确实不会阻塞主线程，不过实测下来当请求很多的时候还是会让页面有一定卡顿的，按队列执行效率又有些低下，所以写了一个简易版的支持worker的compressorjs：
export default async function compressImg(imgFile: File, option: { quality?: number, type?: string }) {
const { quality = 0.5, type = 'image/jpeg' } = option;
const bitmap = await createImageBitmap(imgFile);
const canvas = new OffscreenCanvas(bitmap.width, bitmap.height);
const ctx = canvas.getContext('2d');
canvas.width = bitmap.width;
canvas.height = bitmap.height;
ctx?.drawImage(bitmap, 0, 0, bitmap.width, bitmap.height);
const compressImgData = await canvas.convertToBlob({ type, quality });
return compressImgData.size < imgFile.size ? compressImgData : imgFile;
}
- 为了高效管理worker，写了一个WorkerPool，会动态分配任务给多个Worker，能够动态创建/释放Worker，总之就是动态实现多个Worker之间的负载均衡。为什么本来就有Worker任务队列的情况下还需要这样处理呢？因为每个Worker面对的任务负担是不同的，如果直接把所有任务平均分配了，那就会出现有的Worker完成所有任务后闲置被回收，有的Worker因为一个大任务一直阻塞后面所有任务的执行，最后导致全部任务执行完成的时间延长。具体完整实现如下，有些地方的实现不是最佳实现方式，不过正常情况下不会非常影响性能，就不care了。
- 相比于
thread.js的pool，WorkerPool实现了运行时根据任务负担动态创建/释放Worker。 - 实测（就是前面那个TinyJPEG）中默认配置下相较于单个worker，性能提升约40%，开8个Worker时通过WorkerPool实现负载均衡调度相较于平均分配任务性能提升约20%，当然TinyJPEG项目的计算应该理论上是调用了GPU，对于普通那种吃点CPU的Worker是什么效果我也不懂，不过理论上会有提升吧，会有提升吧？
import { throttle } from "lodash-es";
export default class WorkerPool {
/**
* WorkerPool模式：尽可能创建更多的Worker
*/
public static MODE_MoreWorker = Symbol('MoreWorker');
/**
* WorkerPool模式：尽可能让已有Worker忙
*/
public static MODE_BusyWorker = Symbol('BusyWorker');
constructor(workerConstructor: new () => Worker, option?: {
MaxTaskPerWorker?: number;
MaxWorkerCount?: number;
MinWorkerCount?: number;
Mode?: typeof WorkerPool.MODE_MoreWorker | typeof WorkerPool.MODE_BusyWorker;
}) {
this.workerConstructor = workerConstructor;
this.MaxTaskPerWorker = option?.MaxTaskPerWorker ?? 2;
this.MaxWorkerCount = option?.MaxWorkerCount ?? 4;
this.MinWorkerCount = option?.MinWorkerCount ?? 0;
this.mode = option?.Mode ?? WorkerPool.MODE_MoreWorker;
new Array(this.MinWorkerCount).fill(null).forEach(() => {
this.newWorker();
});
}
private workerConstructor: new () => Worker;
private workerPool: Worker[] = [];
private workerTaskMap: Map<number, { resolve: (...args: any[]) => void, reject: (error: any) => void }> = new Map();
private workerTaskCountMap = new WeakMap<Worker, number>();
private taskRequestBuffer: {
message: { [k: string]: any },
transfer: Transferable[]
}[] = [];
private MaxTaskPerWorker: number;
private MaxWorkerCount: number;
private MinWorkerCount: number;
private mode: typeof WorkerPool.MODE_MoreWorker | typeof WorkerPool.MODE_BusyWorker;
private _idCounter = 0;
private get isWorkerAllBusy() {
return this.workerPool.every(worker => this.workerTaskCountMap.get(worker)! >= this.MaxTaskPerWorker);
}
private get isPoolFull() {
return this.workerPool.length >= this.MaxWorkerCount;
}
private getAvailableWorker() {
if (this.workerPool.length === 0 || (this.mode === WorkerPool.MODE_BusyWorker ? this.isWorkerAllBusy : !this.isPoolFull)) {
if (this.mode === WorkerPool.MODE_BusyWorker && this.isPoolFull) {
return null;
}
this.newWorker();
}
if (this.mode === WorkerPool.MODE_MoreWorker && this.isWorkerAllBusy) {
return null;
}
// 性能差不多行了
this.workerPool.sort((a, b) => {
const countA = this.workerTaskCountMap.get(a)!;
const countB = this.workerTaskCountMap.get(b)!;
return countA - countB;
});
return this.mode === WorkerPool.MODE_BusyWorker
? (this.workerPool.filter(worker => this.workerTaskCountMap.get(worker)! < this.MaxTaskPerWorker).toReversed()[0] || null)
: this.workerPool[0];
}
private newWorker() {
if (this.workerPool.length >= this.MaxWorkerCount) {
return;
}
const newWorker = new this.workerConstructor();
newWorker.onmessage = (event) => {
const { id } = event.data;
const task = this.workerTaskMap.get(id);
if (task) {
task.resolve(event.data);
this.workerTaskMap.delete(id);
if (this.taskRequestBuffer.length > 0) {
const request = this.taskRequestBuffer.shift()!;
newWorker.postMessage(request.message, request.transfer);
// console.log(`WorkerPool: Task from buffer sent to worker, current task request buffer length: ${this.taskRequestBuffer.length}`);
} else {
this.workerTaskCountMap.set(newWorker, this.workerTaskCountMap.get(newWorker)! - 1);
}
} else {
console.error(`WorkerPool: No task found for worker message with id: ${id}`);
}
this.terminateInactivityWorker();
};
newWorker.onerror = (event) => {
const { id } = JSON.parse(event.message)
const task = this.workerTaskMap.get(id);
if (task) {
task.resolve(event.message);
this.workerTaskMap.delete(id);
if (this.taskRequestBuffer.length > 0) {
const request = this.taskRequestBuffer.shift()!;
newWorker.postMessage(request.message, request.transfer);
// console.log(`WorkerPool: Task from buffer sent to worker, current task request buffer length: ${this.taskRequestBuffer.length}`);
} else {
this.workerTaskCountMap.set(newWorker, this.workerTaskCountMap.get(newWorker)! - 1);
}
} else {
console.error(`WorkerPool: No task found for worker message with id: ${id}`);
}
this.terminateInactivityWorker();
};
this.workerPool.push(newWorker);
this.workerTaskCountMap.set(newWorker, 0);
// console.log(`WorkerPool: New worker created, current worker count: ${this.workerPool.length}`);
}
private terminateInactivityWorker = throttle(() => {
if (this.workerPool.length <= this.MinWorkerCount) {
return;
}
const inactivityWorkers = this.workerPool.filter(worker => this.workerTaskCountMap.get(worker) === 0)
for (const worker of inactivityWorkers) {
if (this.workerPool.length <= this.MinWorkerCount) {
break;
}
worker.terminate();
this.workerPool = this.workerPool.filter(w => w !== worker);
this.workerTaskCountMap.delete(worker);
}
// console.log(`WorkerPool: Inactivity workers terminated, current worker count: ${this.workerPool.length}`);
}, 500, {
trailing: true,
});
public terminate() {
this.workerPool.forEach(worker => worker.terminate());
this.workerTaskMap.forEach((task, id) => {
task.reject(new Error(`Worker terminated before task completion: ${id}`));
});
this.workerPool = [];
this.workerTaskMap.clear();
}
public postMessage(message: { [k: string]: any }, transfer?: Transferable[]): Promise<any> {
const id = this._idCounter++;
const worker = this.getAvailableWorker();
return new Promise<any>((resolve, reject) => {
this.workerTaskMap.set(id, { resolve, reject });
if (worker) {
this.workerTaskCountMap.set(worker, this.workerTaskCountMap.get(worker)! + 1);
worker.postMessage({ ...message, id }, transfer ?? []);
} else {
this.taskRequestBuffer.push({ message: { ...message, id }, transfer: transfer ?? [] });
}
});
}
}
- 页面没啥好说的，tailwindcss搭了一个简易小页面。不过压缩图片还是tinypng用途更广泛一些，毕竟png支持透明背景吧。
el-slider水合有问题，懒得管了。使用Worker版本的Compressor搭配WorkPool以后性能表现有极大的提升。
Bobj
-
初衷是做一个将js的对象数据（即
Object）序列化为二进制数据的库，命名Bobj基本就是抄袭Bson的，数据的结构设计上有些参考了mp4文件的box格式。后来发现我的库在事实上几乎就是msgpack的下位替代品。 -
Bobj在设计之初上就支持通过插件拓展对于多种数据对象的序列化/逆序列化能力（天哪简直和
msgpack一样呢），而由此衍生出的序列化算法虽然在序列化普通对象时性能惨不忍睹（大约是msgpackr的十分之一不到），但是在序列化大型U8A对象时（大概1MB）性能表现竟然反超了msgpackr，翻了下msgpackr的代码，发现应该时msgpackr的序列化过程是先创建一个8KB的缓冲区，在容量不够时进行扩容，而对于像是U8A这样的大型数据对象，他的处理逻辑是额外分配.25到.4倍（上限是不是.4忘了，下限是.25）的内存空间。而Bobj的算法是先生成各个部分的二进制数据，最后统一合并到一起，所需的内存空间是确定的，避免了创建缓冲区以及类似分配内存这样的操作，自然不用额外分配内存以应对可能到来的数据。 -
得益于Bobj的序列化器设计的拓展性有点太好了，所以我基于Bobj的序列化器写了一套插件，使得Bobj的序列化器能将对象数据序列化为msgpack数据。性能测试下来依旧是普通对象拉垮，对于U8A性能表现良好。不过用msgpack的大概率不是为了实现JSON能实现的事情啊，所以我的这个东西还是有些可取之处的。
-
bobj提供同步序列化与异步序列化方法，由于异步函数的特性，频繁调用时会产生极大的额外性能开销，故还是同步序列化速度快点。
-
首页的bobj-demo一半是AI Generate的，写的好烂，不过反正也没人看没人用，就无所谓了。
-
首页的自解压也是基于这个项目，基本就是在bobj-demo的基础上小范围魔改实现。自解压程序的实现实际上就是读取程序数据本体后面的通过msgpack序列化的多个文件数据而已，序列化过程使用bobj实现。
日常
- 买彩票买了一张，中了20rmb。
- 拿中奖一半的钱又买了一张，中了50rmb。
- 最后一张关于福大的照片，雨后彩虹。