Improving Model Latency With a Colleague

8/17/2026intermediate

Speakers

You

あなた

Tomasz Nowak

シニア機械学習エンジニア（ポーランド）

この場面について

デプロイ済みのAIモデルのレイテンシが悪化していることに気づき、海外の同僚とビデオ通話で原因を切り分け、改善案を相談する場面。

Summary (JP)

推論APIの応答が遅くなっていることに気づいたあなたは、ポーランド在住のシニアMLエンジニア、トマシュにビデオ通話で相談します。プロファイリングの結果からボトルネックを特定し、キャッシュの導入と量子化という2つの改善案を検討します。最後にそれぞれの担当を決め、金曜日に進捗を共有することで合意します。

会話

Speed:

文をクリックすると、その部分から読み上げが始まります。

You:

Hey Tomasz, do you have a few minutes to talk about the latency issue?

Tomasz Nowak:

Sure, I saw the alert this morning. What's going on?

You:

Our inference API is averaging almost two seconds per request now.

Tomasz Nowak:

That's way above our target. Where's the bottleneck exactly?

You:

I think it's the embedding step before we hit the model.

Tomasz Nowak:

Have you profiled it, or is that just a guess?

You:

I ran a quick profile last night, and it confirmed it.

Tomasz Nowak:

Okay, so what options are we looking at?

You:

We could cache embeddings for repeated queries, that seems low-risk.

Tomasz Nowak:

Good idea. What's our current cache hit rate, do you know?

You:

Around 30%, so there's clearly room to improve it.

Tomasz Nowak:

We could also try quantizing the model to cut latency further.

You:

That's a bigger change though. Might affect accuracy a bit.

Tomasz Nowak:

True, but let's benchmark it before we rule it out.

You:

Agreed. I'll set up the caching layer first, it's quicker.

Tomasz Nowak:

And I'll run the quantization benchmarks in parallel this week.

You:

Sounds like a solid plan. Let's sync again on Friday.

Tomasz Nowak:

Works for me. Thanks for digging into this so fast.

Vocabulary

latency

Meaning: 遅延、応答にかかる時間

Example: The new caching layer cut our latency in half.

bottleneck

Meaning: ボトルネック、処理速度を制限している箇所

Example: The database query turned out to be the real bottleneck.

profile (verb)

Meaning: （コードやシステムの）性能を計測・分析する

Example: We profiled the pipeline to see which step was slowest.

cache hit rate

Meaning: キャッシュヒット率（キャッシュが利用された割合）

Example: Our cache hit rate improved after we added more storage.

quantize

Meaning: （モデルを）量子化する、パラメータの精度を下げて軽量化する

Example: Quantizing the model reduced its size significantly.

benchmark (verb)

Meaning: ベンチマークを取る、性能を測定して比較する

Example: Let's benchmark both approaches before deciding.

low-risk

Meaning: 低リスクの、失敗しても影響が小さい

Example: Adding a cache is a fairly low-risk change.

sync

Meaning: 同期する、情報共有のために集まる

Example: Let's sync again once you have the results.

Quiz

1. What problem does You bring up at the start of the call?

2. Where does You think the bottleneck is?

3. What is the current cache hit rate they mention?

4. What do You and Tomasz decide to do before ruling out quantization?

5. When do they agree to sync again?

← Back to conversations