Llama 3.2 1B Instruct Q40 Benchmarks, 8 x Raspberry Pi 4B 8GB #123

githuba9f5404 · 2024-09-26T04:42:41Z

githuba9f5404
Sep 26, 2024

Running on this hardware setup, overclocked with an average clock speed of 2150 MHZ (standard is 1500 MHZ).

Overall

Avg tokens / second: 6.23
Avg generation time: 160.60 ms
Avg inference time:  149.70 ms
Avg transfer time:   8.95 ms

Run 1

sudo nice -n -20 ../../dllama inference --model dllama_model_llama3_2_1b_instruct_q40.m --tokenizer dllama_tokenizer_llama3_2_1b_instruct_q40.t --buffer-float-type q80 --prompt "The Eiffel Tower is" --steps 64 --nthreads 4 --workers 192.168.60.11:9998 192.168.60.12:9998 192.168.60.13:9998 192.168.60.14:9998 192.168.60.15:9998 192.168.60.16:9998 192.168.60.17:9998
💡 arch: llama
💡 hiddenAct: silu
💡 dim: 2048
💡 hiddenDim: 8192
💡 nLayers: 16
💡 nHeads: 32
💡 nKvHeads: 8
💡 vocabSize: 128256
💡 seqLen: 8192
💡 nSlices: 8
💡 ropeTheta: 500000.0
📄 bosId: 128000
📄 eosId: 128001
📄 chatEosId: 128009
🕒 ropeCacheSize: 8192 kB
⏩ Loaded 1705128 kB
🔶 G  184 ms I  147 ms T   37 ms S 468188 kB R    476 kB The
🔶 G  185 ms I  169 ms T   16 ms S    476 kB R    476 kB  E
🔶 G  156 ms I  151 ms T    4 ms S    476 kB R    476 kB iff
🔶 G  153 ms I  147 ms T    6 ms S    476 kB R    476 kB el
🔶 G  174 ms I  171 ms T    3 ms S    476 kB R    476 kB  Tower
🔶 G  153 ms I  152 ms T    1 ms S    476 kB R    476 kB  is
🔶 G  156 ms I  151 ms T    3 ms S    476 kB R    476 kB  an
🔶 G  155 ms I  149 ms T    4 ms S    476 kB R    476 kB  iconic
🔶 G  158 ms I  150 ms T    5 ms S    476 kB R    476 kB  symbol
🔶 G  155 ms I  147 ms T    6 ms S    476 kB R    476 kB  of
🔶 G  175 ms I  165 ms T    8 ms S    476 kB R    476 kB  Paris
🔶 G  155 ms I  147 ms T    6 ms S    476 kB R    476 kB ,
🔶 G  155 ms I  147 ms T    6 ms S    476 kB R    476 kB  France
🔶 G  155 ms I  145 ms T    8 ms S    476 kB R    476 kB ,
🔶 G  155 ms I  146 ms T    7 ms S    476 kB R    476 kB  and
🔶 G  159 ms I  151 ms T    6 ms S    476 kB R    476 kB  one
🔶 G  178 ms I  167 ms T    8 ms S    476 kB R    476 kB  of
🔶 G  156 ms I  148 ms T    6 ms S    476 kB R    476 kB  the
🔶 G  155 ms I  148 ms T    5 ms S    476 kB R    476 kB  most
🔶 G  156 ms I  148 ms T    6 ms S    476 kB R    476 kB  recognizable
🔶 G  156 ms I  150 ms T    4 ms S    476 kB R    476 kB  landmarks
🔶 G  167 ms I  159 ms T    6 ms S    476 kB R    476 kB  in
🔶 G  214 ms I  197 ms T   15 ms S    476 kB R    476 kB  the
🔶 G  158 ms I  148 ms T    8 ms S    476 kB R    476 kB  world
🔶 G  156 ms I  144 ms T   10 ms S    476 kB R    476 kB .
🔶 G  155 ms I  147 ms T    6 ms S    476 kB R    476 kB  The
🔶 G  157 ms I  146 ms T    8 ms S    476 kB R    476 kB  E
🔶 G  158 ms I  150 ms T    6 ms S    476 kB R    476 kB iff
🔶 G  175 ms I  166 ms T    7 ms S    476 kB R    476 kB el
🔶 G  157 ms I  150 ms T    5 ms S    476 kB R    476 kB  Tower
🔶 G  156 ms I  148 ms T    6 ms S    476 kB R    476 kB  has
🔶 G  156 ms I  148 ms T    5 ms S    476 kB R    476 kB  been
🔶 G  156 ms I  147 ms T    7 ms S    476 kB R    476 kB  a
🔶 G  158 ms I  149 ms T    7 ms S    476 kB R    476 kB  popular
🔶 G  156 ms I  148 ms T    6 ms S    476 kB R    476 kB  destination
🔶 G  177 ms I  159 ms T   16 ms S    476 kB R    476 kB  for
🔶 G  155 ms I  148 ms T    5 ms S    476 kB R    476 kB  tourists
🔶 G  156 ms I  146 ms T    8 ms S    476 kB R    476 kB  and
🔶 G  156 ms I  149 ms T    5 ms S    476 kB R    476 kB  locals
🔶 G  157 ms I  150 ms T    5 ms S    476 kB R    476 kB  alike
🔶 G  156 ms I  147 ms T    7 ms S    476 kB R    476 kB  for
🔶 G  181 ms I  171 ms T    8 ms S    476 kB R    476 kB  over
🔶 G  156 ms I  150 ms T    4 ms S    476 kB R    476 kB  a
🔶 G  155 ms I  149 ms T    4 ms S    476 kB R    476 kB  century
🔶 G  156 ms I  147 ms T    7 ms S    476 kB R    476 kB ,
🔶 G  156 ms I  149 ms T    5 ms S    476 kB R    476 kB  and
🔶 G  159 ms I  152 ms T    5 ms S    476 kB R    476 kB  its
🔶 G  176 ms I  172 ms T    2 ms S    476 kB R    476 kB  appeal
🔶 G  158 ms I  149 ms T    7 ms S    476 kB R    476 kB  is
🔶 G  156 ms I  147 ms T    6 ms S    476 kB R    476 kB  undeniable
🔶 G  156 ms I  149 ms T    5 ms S    476 kB R    476 kB .


🔶 G  156 ms I  146 ms T    7 ms S    476 kB R    476 kB But
🔶 G  166 ms I  158 ms T    6 ms S    476 kB R    476 kB  have
🔶 G  183 ms I  173 ms T    8 ms S    476 kB R    476 kB  you
🔶 G  157 ms I  151 ms T    4 ms S    476 kB R    476 kB  ever
🔶 G  155 ms I  146 ms T    7 ms S    476 kB R    476 kB  wondered
🔶 G  156 ms I  145 ms T    9 ms S    476 kB R    476 kB  about
🔶 G  156 ms I  147 ms T    7 ms S    476 kB R    476 kB  the
🔶 G  158 ms I  151 ms T    5 ms S    476 kB R    476 kB  E
🔶 G  157 ms I  151 ms T    4 ms S    476 kB R    476 kB iff
🔶 G  176 ms I  168 ms T    6 ms S    476 kB R    476 kB el
🔶 G  156 ms I  151 ms T    3 ms S    476 kB R    476 kB  Tower
🔶 G  157 ms I  149 ms T    6 ms S    476 kB R    476 kB 's
🔶 G  156 ms I  150 ms T    4 ms S    476 kB R    476 kB  history
Generated tokens:    64
Avg tokens / second: 6.20
Avg generation time: 161.22 ms
Avg inference time:  152.55 ms
Avg transfer time:   6.75 ms

Run 2

$ sudo nice -n -20 ../../dllama inference --model dllama_model_llama3_2_1b_instruct_q40.m --tokenizer dllama_tokenizer_llama3_2_1b_instruct_q40.t --buffer-float-type q80 --prompt "The Eiffel Tower is" --steps 64 --nthreads 4 --workers 192.168.60.11:9998 192.168.60.12:9998 192.168.60.13:9998 192.168.60.14:9998 192.168.60.15:9998 192.168.60.16:9998 192.168.60.17:9998
💡 arch: llama
💡 hiddenAct: silu
💡 dim: 2048
💡 hiddenDim: 8192
💡 nLayers: 16
💡 nHeads: 32
💡 nKvHeads: 8
💡 vocabSize: 128256
💡 seqLen: 8192
💡 nSlices: 8
💡 ropeTheta: 500000.0
📄 bosId: 128000
📄 eosId: 128001
📄 chatEosId: 128009
🕒 ropeCacheSize: 8192 kB
⏩ Loaded 1705128 kB
🔶 G  187 ms I  140 ms T   46 ms S 468188 kB R    476 kB The
🔶 G  150 ms I  142 ms T    8 ms S    476 kB R    476 kB  E
🔶 G  152 ms I  143 ms T    9 ms S    476 kB R    476 kB iff
🔶 G  150 ms I  144 ms T    6 ms S    476 kB R    476 kB el
🔶 G  150 ms I  142 ms T    8 ms S    476 kB R    476 kB  Tower
🔶 G  151 ms I  145 ms T    6 ms S    476 kB R    476 kB  is
🔶 G  174 ms I  163 ms T    8 ms S    476 kB R    476 kB  
🔶 G  153 ms I  144 ms T    7 ms S    476 kB R    476 kB 324
🔶 G  153 ms I  145 ms T    6 ms S    476 kB R    476 kB  meters
🔶 G  154 ms I  146 ms T    6 ms S    476 kB R    476 kB  (
🔶 G  151 ms I  143 ms T    6 ms S    476 kB R    476 kB 1
🔶 G  152 ms I  144 ms T    6 ms S    476 kB R    476 kB ,
🔶 G  169 ms I  161 ms T    6 ms S    476 kB R    476 kB 063
🔶 G  153 ms I  142 ms T    9 ms S    476 kB R    476 kB  feet
🔶 G  152 ms I  140 ms T    9 ms S    476 kB R    476 kB )
🔶 G  154 ms I  146 ms T    6 ms S    476 kB R    476 kB  tall
🔶 G  151 ms I  140 ms T    9 ms S    476 kB R    476 kB .
🔶 G  162 ms I  152 ms T    8 ms S    476 kB R    476 kB  If
🔶 G  168 ms I  155 ms T   10 ms S    476 kB R    476 kB  a
🔶 G  156 ms I  146 ms T    7 ms S    476 kB R    476 kB  
🔶 G  152 ms I  136 ms T   14 ms S    476 kB R    476 kB 300
🔶 G  155 ms I  147 ms T    6 ms S    476 kB R    476 kB -w
🔶 G  153 ms I  144 ms T    7 ms S    476 kB R    476 kB att
🔶 G  152 ms I  141 ms T    9 ms S    476 kB R    476 kB  radio
🔶 G  152 ms I  141 ms T    9 ms S    476 kB R    476 kB  transmitter
🔶 G  166 ms I  156 ms T    8 ms S    476 kB R    476 kB  and
🔶 G  188 ms I  170 ms T   16 ms S    476 kB R    476 kB  a
🔶 G  153 ms I  146 ms T    5 ms S    476 kB R    476 kB  
🔶 G  154 ms I  146 ms T    6 ms S    476 kB R    476 kB 200
🔶 G  153 ms I  142 ms T    9 ms S    476 kB R    476 kB -w
🔶 G  151 ms I  141 ms T    8 ms S    476 kB R    476 kB att
🔶 G  168 ms I  164 ms T    2 ms S    476 kB R    476 kB  radio
🔶 G  151 ms I  142 ms T    7 ms S    476 kB R    476 kB  receiver
🔶 G  153 ms I  141 ms T   10 ms S    476 kB R    476 kB  are
🔶 G  154 ms I  140 ms T   12 ms S    476 kB R    476 kB  connected
🔶 G  193 ms I  176 ms T   15 ms S    476 kB R    476 kB  in
🔶 G  152 ms I  142 ms T    8 ms S    476 kB R    476 kB  series
🔶 G  168 ms I  162 ms T    4 ms S    476 kB R    476 kB  to
🔶 G  154 ms I  143 ms T    8 ms S    476 kB R    476 kB  receive
🔶 G  152 ms I  144 ms T    6 ms S    476 kB R    476 kB  the
🔶 G  156 ms I  143 ms T   10 ms S    476 kB R    476 kB  same
🔶 G  152 ms I  142 ms T    8 ms S    476 kB R    476 kB  signal
🔶 G  153 ms I  143 ms T    7 ms S    476 kB R    476 kB ,
🔶 G  152 ms I  142 ms T    8 ms S    476 kB R    476 kB  what
🔶 G  172 ms I  162 ms T    8 ms S    476 kB R    476 kB  is
🔶 G  152 ms I  141 ms T    9 ms S    476 kB R    476 kB  the
🔶 G  152 ms I  141 ms T    9 ms S    476 kB R    476 kB  total
🔶 G  157 ms I  150 ms T    5 ms S    476 kB R    476 kB  power
🔶 G  152 ms I  141 ms T    9 ms S    476 kB R    476 kB  in
🔶 G  162 ms I  152 ms T    8 ms S    476 kB R    476 kB  watts
🔶 G  205 ms I  192 ms T   11 ms S    476 kB R    476 kB ?

🔶 G  154 ms I  146 ms T    6 ms S    476 kB R    476 kB ##
🔶 G  152 ms I  144 ms T    6 ms S    476 kB R    476 kB  Step
🔶 G  154 ms I  143 ms T    9 ms S    476 kB R    476 kB  
🔶 G  151 ms I  142 ms T    7 ms S    476 kB R    476 kB 1
🔶 G  153 ms I  148 ms T    3 ms S    476 kB R    476 kB :
🔶 G  167 ms I  160 ms T    5 ms S    476 kB R    476 kB  Determine
🔶 G  154 ms I  148 ms T    4 ms S    476 kB R    476 kB  the
🔶 G  153 ms I  142 ms T    9 ms S    476 kB R    476 kB  current
🔶 G  155 ms I  146 ms T    7 ms S    476 kB R    476 kB  in
🔶 G  152 ms I  144 ms T    6 ms S    476 kB R    476 kB  the
🔶 G  152 ms I  142 ms T    8 ms S    476 kB R    476 kB  system
🔶 G  152 ms I  143 ms T    7 ms S    476 kB R    476 kB 

🔶 G  175 ms I  164 ms T    9 ms S    476 kB R    476 kB To
Generated tokens:    64
Avg tokens / second: 6.33
Avg generation time: 157.89 ms
Avg inference time:  147.62 ms
Avg transfer time:   8.33 ms

Run 3

$ sudo nice -n -20 ../../dllama inference --model dllama_model_llama3_2_1b_instruct_q40.m --tokenizer dllama_tokenizer_llama3_2_1b_instruct_q40.t --buffer-float-type q80 --prompt "The Eiffel Tower is" --steps 64 --nthreads 4 --workers 192.168.60.11:9998 192.168.60.12:9998 192.168.60.13:9998 192.168.60.14:9998 192.168.60.15:9998 192.168.60.16:9998 192.168.60.17:9998
💡 arch: llama
💡 hiddenAct: silu
💡 dim: 2048
💡 hiddenDim: 8192
💡 nLayers: 16
💡 nHeads: 32
💡 nKvHeads: 8
💡 vocabSize: 128256
💡 seqLen: 8192
💡 nSlices: 8
💡 ropeTheta: 500000.0
📄 bosId: 128000
📄 eosId: 128001
📄 chatEosId: 128009
🕒 ropeCacheSize: 8192 kB
⏩ Loaded 1705128 kB
🔶 G  209 ms I  164 ms T   45 ms S 468188 kB R    476 kB The
🔶 G  150 ms I  143 ms T    7 ms S    476 kB R    476 kB  E
🔶 G  151 ms I  140 ms T   11 ms S    476 kB R    476 kB iff
🔶 G  150 ms I  142 ms T    8 ms S    476 kB R    476 kB el
🔶 G  150 ms I  142 ms T    8 ms S    476 kB R    476 kB  Tower
🔶 G  151 ms I  140 ms T   11 ms S    476 kB R    476 kB  is
🔶 G  171 ms I  156 ms T   13 ms S    476 kB R    476 kB  a
🔶 G  159 ms I  148 ms T    8 ms S    476 kB R    476 kB  must
🔶 G  234 ms I  206 ms T   26 ms S    476 kB R    476 kB -
🔶 G  153 ms I  137 ms T   14 ms S    476 kB R    476 kB see
🔶 G  152 ms I  140 ms T   10 ms S    476 kB R    476 kB  attraction
🔶 G  153 ms I  141 ms T   10 ms S    476 kB R    476 kB ,
🔶 G  173 ms I  160 ms T   11 ms S    476 kB R    476 kB  especially
🔶 G  156 ms I  145 ms T    9 ms S    476 kB R    476 kB  at
🔶 G  152 ms I  141 ms T    9 ms S    476 kB R    476 kB  night
🔶 G  154 ms I  142 ms T   10 ms S    476 kB R    476 kB  when
🔶 G  153 ms I  138 ms T   13 ms S    476 kB R    476 kB  it
🔶 G  152 ms I  141 ms T    9 ms S    476 kB R    476 kB  is
🔶 G  154 ms I  141 ms T    9 ms S    476 kB R    476 kB  illuminated
🔶 G  175 ms I  162 ms T   11 ms S    476 kB R    476 kB .
🔶 G  152 ms I  142 ms T    8 ms S    476 kB R    476 kB  The
🔶 G  153 ms I  142 ms T    9 ms S    476 kB R    476 kB  E
🔶 G  153 ms I  140 ms T   10 ms S    476 kB R    476 kB iff
🔶 G  154 ms I  139 ms T   13 ms S    476 kB R    476 kB el
🔶 G  152 ms I  141 ms T    9 ms S    476 kB R    476 kB  Tower
🔶 G  178 ms I  165 ms T   10 ms S    476 kB R    476 kB  is
🔶 G  185 ms I  167 ms T   16 ms S    476 kB R    476 kB  an
🔶 G  153 ms I  143 ms T    8 ms S    476 kB R    476 kB  iconic
🔶 G  153 ms I  140 ms T   10 ms S    476 kB R    476 kB  symbol
🔶 G  153 ms I  139 ms T   12 ms S    476 kB R    476 kB  of
🔶 G  162 ms I  152 ms T    8 ms S    476 kB R    476 kB  France
🔶 G  220 ms I  199 ms T   19 ms S    476 kB R    476 kB ,
🔶 G  189 ms I  169 ms T   18 ms S    476 kB R    476 kB  with
🔶 G  155 ms I  142 ms T   11 ms S    476 kB R    476 kB  over
🔶 G  152 ms I  141 ms T    9 ms S    476 kB R    476 kB  
🔶 G  153 ms I  140 ms T   11 ms S    476 kB R    476 kB 1
🔶 G  152 ms I  140 ms T   10 ms S    476 kB R    476 kB  million
🔶 G  178 ms I  161 ms T   15 ms S    476 kB R    476 kB  visitors
🔶 G  153 ms I  141 ms T   10 ms S    476 kB R    476 kB  each
🔶 G  153 ms I  143 ms T    8 ms S    476 kB R    476 kB  year
🔶 G  153 ms I  140 ms T   11 ms S    476 kB R    476 kB .
🔶 G  153 ms I  141 ms T   10 ms S    476 kB R    476 kB  A
🔶 G  154 ms I  139 ms T   12 ms S    476 kB R    476 kB  visit
🔶 G  173 ms I  164 ms T    7 ms S    476 kB R    476 kB  to
🔶 G  155 ms I  144 ms T    9 ms S    476 kB R    476 kB  Paris
🔶 G  153 ms I  139 ms T   12 ms S    476 kB R    476 kB ,
🔶 G  153 ms I  138 ms T   12 ms S    476 kB R    476 kB  the
🔶 G  153 ms I  138 ms T   13 ms S    476 kB R    476 kB  city
🔶 G  153 ms I  138 ms T   13 ms S    476 kB R    476 kB  where
🔶 G  153 ms I  145 ms T    6 ms S    476 kB R    476 kB  the
🔶 G  180 ms I  166 ms T   11 ms S    476 kB R    476 kB  E
🔶 G  190 ms I  170 ms T   18 ms S    476 kB R    476 kB iff
🔶 G  153 ms I  140 ms T   11 ms S    476 kB R    476 kB el
🔶 G  153 ms I  142 ms T    9 ms S    476 kB R    476 kB  Tower
🔶 G  152 ms I  138 ms T   12 ms S    476 kB R    476 kB  is
🔶 G  153 ms I  143 ms T    8 ms S    476 kB R    476 kB  located
🔶 G  175 ms I  162 ms T   11 ms S    476 kB R    476 kB ,
🔶 G  193 ms I  171 ms T   20 ms S    476 kB R    476 kB  is
🔶 G  153 ms I  142 ms T    9 ms S    476 kB R    476 kB  a
🔶 G  155 ms I  140 ms T   13 ms S    476 kB R    476 kB  must
🔶 G  153 ms I  140 ms T   11 ms S    476 kB R    476 kB -do
🔶 G  176 ms I  167 ms T    7 ms S    476 kB R    476 kB  experience
🔶 G  175 ms I  160 ms T   13 ms S    476 kB R    476 kB .


🔶 G  193 ms I  170 ms T   20 ms S    476 kB R    476 kB Vis
Generated tokens:    64
Avg tokens / second: 6.15
Avg generation time: 162.69 ms
Avg inference time:  148.94 ms
Avg transfer time:   11.78 ms

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Llama 3.2 1B Instruct Q40 Benchmarks, 8 x Raspberry Pi 4B 8GB #123

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 0 comments

Select a reply

Llama 3.2 1B Instruct Q40 Benchmarks, 8 x Raspberry Pi 4B 8GB #123

githuba9f5404 Sep 26, 2024

Replies: 0 comments

githuba9f5404
Sep 26, 2024