NFS performances

[Emmanuel Dreyfus]

Hello

I have a NFS setup with both NetBSD 6 client and server over a
gigabit network. Theperformance seems week, even whle client,
server and network are almost idle.

The test: time dd if=/dev/zero of=test bs=1024k count=100

Done on the NFS server itself:
5.31s real 0.00s user 0.39s system

Done over NFS:
9.82s real 0.00s user 0.12s system

The overhead looks huge. This is a UDP mount with a 1500 bytes MTU,
ping is at 0.8 ms from a virtualized client, ang 0.28 ms for a physical one.

Are the numbers reasonable? Should I consider a 59% NFS overhead as
acceptable, or are there some parameters to tweak?

[J. Lewis Muir]

Hello
I have a NFS setup with both NetBSD 6 client and server over a gigabit
network. Theperformance seems week, even whle client, server and
network are almost idle.
The test: time dd if=/dev/zero of=test bs=1024k count=100
5.31s real 0.00s user 0.39s system
9.82s real 0.00s user 0.12s system
The overhead looks huge. This is a UDP mount with a 1500 bytes MTU,
ping is at 0.8 ms from a virtualized client, ang 0.28 ms for a
physical one.
Are the numbers reasonable? Should I consider a 59% NFS overhead as
acceptable, or are there some parameters to tweak?

Hi, Emmanuel.

I have no idea if it will help, but have you tried adjusting
the kernel's UDP send and receive buffer sizes (i.e., for IPv4,
net.inet.udp.sendspace and net.inet.udp.recvspace) with sysctl?

Lewis

[Jan Stary]

Hello
I have a NFS setup with both NetBSD 6 client and server over a gigabit
network. Theperformance seems week, even whle client, server and
network are almost idle.
The test: time dd if=/dev/zero of=test bs=1024k count=100
5.31s real 0.00s user 0.39s system
9.82s real 0.00s user 0.12s system

What exactly does "over NFS" mean here? The client's
/dev/zero gets written to server:/mnt/file, right?

Maybe I am missing something obvious, but in the first run,
nothing is sent over the network, and in the second run,
100m is sent over the network.

[Emmanuel Dreyfus]

Post by J. Lewis Muir
I have no idea if it will help, but have you tried adjusting
the kernel's UDP send and receive buffer sizes (i.e., for IPv4,
net.inet.udp.sendspace and net.inet.udp.recvspace) with sysctl?

I just tried, and it does not seems to be the thing that holds
performance down, a raise from 9216 to 655360 did not change anything.

[J. Lewis Muir]

Hello
I have a NFS setup with both NetBSD 6 client and server over a gigabit
network. Theperformance seems week, even whle client, server and
network are almost idle.
The test: time dd if=/dev/zero of=test bs=1024k count=100
5.31s real 0.00s user 0.39s system
9.82s real 0.00s user 0.12s system
The overhead looks huge. This is a UDP mount with a 1500 bytes MTU,
ping is at 0.8 ms from a virtualized client, ang 0.28 ms for a
physical one.
Are the numbers reasonable? Should I consider a 59% NFS overhead as
acceptable, or are there some parameters to tweak?

Hi, Emmanuel.

RHEL 5 machine writing to an NFS v3 mount over GigE (1500 MTU):
1.22s real 0.00s user 0.06s system

That's way faster than even your local test. So, I'd say your numbers
do not seem reasonable.

Have you already determined that the problem is with NFS, not the
network (e.g. by using netperf or maybe just "time dd if=/dev/zero
bs=1024k count=100 | ssh NFS_SERVER 'cat > test'")?

Lewis

[Jan Stary]

Post by J. Lewis Muir

Hello
I have a NFS setup with both NetBSD 6 client and server over a gigabit
network. Theperformance seems week, even whle client, server and
network are almost idle.
The test: time dd if=/dev/zero of=test bs=1024k count=100
5.31s real 0.00s user 0.39s system
9.82s real 0.00s user 0.12s system
The overhead looks huge. This is a UDP mount with a 1500 bytes MTU,
ping is at 0.8 ms from a virtualized client, ang 0.28 ms for a physical one.
Are the numbers reasonable? Should I consider a 59% NFS overhead as
acceptable, or are there some parameters to tweak?

Hi, Emmanuel.
1.22s real 0.00s user 0.06s system
That's way faster than even your local test. So, I'd say your numbers
do not seem reasonable.

We don't have any kind of detail for this comparison to even make sense.
How exactly is the filesystem exported? How exactly is it mounted?
In particular, is it async on linux? Is it even the same server?

Post by J. Lewis Muir
Have you already determined that the problem is with NFS,
not the network

What "problem"? I still don't get it: writing over a network
is considerably slower than _not_ writing over a network, sure.

Post by J. Lewis Muir
(e.g. by using netperf or maybe just "time dd if=/dev/zero
bs=1024k count=100 | ssh NFS_SERVER 'cat > test'")?

Ech, this brings irelevant influences into your "measurement";
for example, how much of that time is spent by ssh encrypting
and compressing the data (which has nothing to do with
your network performance)?

Jan

[J. Lewis Muir]

Post by Jan Stary

Post by J. Lewis Muir

Post by Emmanuel Dreyfus
Hello
I have a NFS setup with both NetBSD 6 client and server over a
gigabit network. Theperformance seems week, even whle client, server
and network are almost idle.
The test: time dd if=/dev/zero of=test bs=1024k count=100
5.31s real 0.00s user 0.39s system
9.82s real 0.00s user 0.12s system
The overhead looks huge. This is a UDP mount with a 1500 bytes MTU,
ping is at 0.8 ms from a virtualized client, ang 0.28 ms for a physical one.
Are the numbers reasonable? Should I consider a 59% NFS overhead as
acceptable, or are there some parameters to tweak?

Hi, Emmanuel.
1.22s real 0.00s user 0.06s system
That's way faster than even your local test. So, I'd say your
numbers do not seem reasonable.

We don't have any kind of detail for this comparison to even make sense.

Hi, Jan.

True, I was just trying to give something for comparison under the
assumption that something was better than nothing.

Post by Jan Stary
How exactly is the filesystem exported?

With the following options:

rw,no_subtree_check,sync

Post by Jan Stary
How exactly is it mounted?

With the following options:

retry=1,bg,intr,noatime,nodev,nosuid,timeo=20

Post by Jan Stary
In particular, is it async on linux?

It is not async, and both the NFS client and server are on Linux.

Post by Jan Stary
Is it even the same server?

I'm not sure what you mean. Do you mean is the NFS server running
NetBSD 6 and the same NFS code as Emmanuel's? No, the server is running
on Linux. But I was just trying to give a ballpark comparison showing
that Emmanuel's setup wasn't even close, thus suggesting something
wasn't right about it.

Post by Jan Stary

Post by J. Lewis Muir
Have you already determined that the problem is with NFS, not the
network

What "problem"? I still don't get it: writing over a network is
considerably slower than _not_ writing over a network, sure.

Post by J. Lewis Muir
(e.g. by using netperf or maybe just "time dd if=/dev/zero bs=1024k
count=100 | ssh NFS_SERVER 'cat > test'")?

Ech, this brings irelevant influences into your "measurement";
for example, how much of that time is spent by ssh encrypting and
compressing the data (which has nothing to do with your network
performance)?

That's true. Note that I suggested netperf first. Still, the
dd-over-ssh thing was just a quick thing to try. I wouldn't think
encrypting the data would be that hard on the machine, and if the
dd-over-ssh thing was grossly faster, it would suggest that something
was definitely wrong with the NFS setup.

Lewis

[Jan Stary]

Post by J. Lewis Muir

Post by Jan Stary

Post by J. Lewis Muir

Post by Emmanuel Dreyfus
Hello
I have a NFS setup with both NetBSD 6 client and server over a
gigabit network. Theperformance seems week, even whle client, server
and network are almost idle.
The test: time dd if=/dev/zero of=test bs=1024k count=100
5.31s real 0.00s user 0.39s system
9.82s real 0.00s user 0.12s system
The overhead looks huge. This is a UDP mount with a 1500 bytes MTU,
ping is at 0.8 ms from a virtualized client, ang 0.28 ms for a physical one.
Are the numbers reasonable? Should I consider a 59% NFS overhead as
acceptable, or are there some parameters to tweak?

Hi, Emmanuel.
1.22s real 0.00s user 0.06s system
That's way faster than even your local test. So, I'd say your
numbers do not seem reasonable.

We don't have any kind of detail for this comparison to even make sense.

Hi, Jan.
True, I was just trying to give something for comparison under the
assumption that something was better than nothing.

Post by Jan Stary
How exactly is the filesystem exported?

rw,no_subtree_check,sync

Post by Jan Stary
How exactly is it mounted?

retry=1,bg,intr,noatime,nodev,nosuid,timeo=20

Post by Jan Stary
In particular, is it async on linux?

It is not async, and both the NFS client and server are on Linux.

Ah, so it has absolutely nothing to do
with the NetBSD NFS server or the NetBSD NFS client.

Post by J. Lewis Muir

Post by Jan Stary
Is it even the same server?

I'm not sure what you mean. Do you mean is the NFS server running
NetBSD 6 and the same NFS code as Emmanuel's? No, the server is running
on Linux. But I was just trying to give a ballpark comparison showing
that Emmanuel's setup wasn't even close, thus suggesting something
wasn't right about it.

We don't know _anything_ about his server, his client,
your server, and your client. This comparison is empty.

Post by J. Lewis Muir

Post by Jan Stary

Post by J. Lewis Muir
Have you already determined that the problem is with NFS, not the
network

What "problem"? I still don't get it: writing over a network is
considerably slower than _not_ writing over a network, sure.

Post by J. Lewis Muir
(e.g. by using netperf or maybe just "time dd if=/dev/zero bs=1024k
count=100 | ssh NFS_SERVER 'cat > test'")?

Ech, this brings irelevant influences into your "measurement";
for example, how much of that time is spent by ssh encrypting and
compressing the data (which has nothing to do with your network
performance)?

That's true. Note that I suggested netperf first. Still, the
dd-over-ssh thing was just a quick thing to try. I wouldn't think
encrypting the data would be that hard on the machine, and if the
dd-over-ssh thing was grossly faster, it would suggest that something
was definitely wrong with the NFS setup.
Lewis

[Eric Haszlakiewicz]

Post by Jan Stary
What "problem"? I still don't get it: writing over a network
is considerably slower than _not_ writing over a network, sure.

Intuitions and rules of thumb regarding performance need to be updated with modern technology in mind:

100MB/9.81sec = ~10.2 MB/s
vs
1Gb/s / 8 b/B = 128MB/s peak throughput not counting any overhead

Just based on raw speed, writing over the network should NOT be slower, so it's pretty obvious there is something not quite right.


Eric

[Emmanuel Dreyfus]

Post by J. Lewis Muir
Have you already determined that the problem is with NFS, not the
network (e.g. by using netperf or maybe just "time dd if=/dev/zero
bs=1024k count=100 | ssh NFS_SERVER 'cat > test'")?

You spoted it, it is not a NFS problem:
10.73s real 1.47s user 0.26s system

[Manuel Bouyer]

Post by Emmanuel Dreyfus
Hello
I have a NFS setup with both NetBSD 6 client and server over a
gigabit network. Theperformance seems week, even whle client,
server and network are almost idle.
The test: time dd if=/dev/zero of=test bs=1024k count=100
5.31s real 0.00s user 0.39s system
9.82s real 0.00s user 0.12s system
The overhead looks huge. This is a UDP mount with a 1500 bytes MTU,
ping is at 0.8 ms from a virtualized client, ang 0.28 ms for a physical one.
Are the numbers reasonable? Should I consider a 59% NFS overhead as
acceptable, or are there some parameters to tweak?

I find the server quite slow on the local test. Does it have enough CPU
to process NFS traffic at this speed ? Have you enabled checksum offloads
on the interfaces ?

Have you tried TCP and UDP mounts, and playing with the -r and -w options ?

[Emmanuel Dreyfus]

Post by Manuel Bouyer

Post by Emmanuel Dreyfus
5.31s real 0.00s user 0.39s system

(...)

Post by Manuel Bouyer
I find the server quite slow on the local test. Does it have enough CPU
to process NFS traffic at this speed ?

Right, this is not a NFS performance problem, but a RAIDframe performance
problem. Here is what I get on a non RAID partition:
# time dd if=/dev/zero of=/var/crash/z bs=1024k count=100
1.63s real 0.00s user 0.34s system

And here is on a RAID backed partition:
# time dd if=/dev/zero of=/nc-boot/z bs=1024k count=100 ;
5.77s real 0.00s user 0.37s system

Here is the RAID setup, note that it uses the same disk where
performance are decent:
# mount
/dev/ld1e on /var/crash type ffs (log, local)
/dev/raid1b on /nc-boot type ffs (log, nodev, nosuid, NFS exported, local)
# raidctl -s raid1
Components:
/dev/ld1a: optimal
/dev/ld2a: optimal
No spares.
Component label for /dev/ld1a:
Row: 0, Column: 0, Num Rows: 1, Num Columns: 2
Version: 2, Serial Number: 2147483647, Mod Counter: 6677
Clean: No, Status: 0
sectPerSU: 128, SUsPerPU: 1, SUsPerRU: 1
Queue size: 100, blocksize: 512, numBlocks: 488259200
RAID Level: 1
Autoconfig: Yes
Root partition: No
Last configured as: raid1
Component label for /dev/ld2a:
Row: 0, Column: 1, Num Rows: 1, Num Columns: 2
Version: 2, Serial Number: 2147483647, Mod Counter: 6677
Clean: No, Status: 0
sectPerSU: 128, SUsPerPU: 1, SUsPerRU: 1
Queue size: 100, blocksize: 512, numBlocks: 488259200
RAID Level: 1
Autoconfig: Yes
Root partition: No
Last configured as: raid1
Parity status: clean
Reconstruction is 100% complete.
Parity Re-write is 100% complete.
Copyback is 100% complete.

[Emmanuel Dreyfus]

Post by Emmanuel Dreyfus
Right, this is not a NFS performance problem, but a RAIDframe performance
probleme [RAIDframe introduces a 300% performance hit]

I found this:
https://zhadum.org.uk/2008/07/25/raid-and-file-system-performance-tuning/

The situaton is the best case: 128 sector per stripe, 16kB block size, 2kB
fragment size.

I also found this:
https://mail-index.netbsd.org/tech-kern/2012/11/13/msg014388.html

But remounting without -o log does not improve the situation. It looks
like if RAIDframe is the only culprit.

[Manuel Bouyer]

Post by Emmanuel Dreyfus

Post by Emmanuel Dreyfus
Right, this is not a NFS performance problem, but a RAIDframe performance
probleme [RAIDframe introduces a 300% performance hit]

https://zhadum.org.uk/2008/07/25/raid-and-file-system-performance-tuning/
The situaton is the best case: 128 sector per stripe, 16kB block size, 2kB
fragment size.
https://mail-index.netbsd.org/tech-kern/2012/11/13/msg014388.html
But remounting without -o log does not improve the situation. It looks
like if RAIDframe is the only culprit.

You should also check the partition alignements.
Also, you didn't say what the underlying hardware is.

[Emmanuel Dreyfus]

Post by Manuel Bouyer
You should also check the partition alignements.

raid1 uses ld1a and ld2a
Both ld1a and ld2a have a NetBSD RAID partition starting at sector 63
(very usual stuff). raid1 has a NetBSD FFS partition starting at
sector 0.

Post by Manuel Bouyer
Also, you didn't say what the underlying hardware is.

SATA disks configured as JBOD (each disk is exported as is) on a
3ware 9550SX controler.

But as said earlier, for a disk that has a partition in a RAID
an another partition outside of the RAID, the performance are weak
with the RAID and good without it. The underlying hardward hence
seems fine.

[Manuel Bouyer]

Post by Emmanuel Dreyfus

Post by Manuel Bouyer
You should also check the partition alignements.

raid1 uses ld1a and ld2a
Both ld1a and ld2a have a NetBSD RAID partition starting at sector 63
(very usual stuff).

For large disks this can be a problem. Try to make is start at 2048.

Post by Emmanuel Dreyfus
raid1 has a NetBSD FFS partition starting at
sector 0.

Post by Manuel Bouyer
Also, you didn't say what the underlying hardware is.

SATA disks configured as JBOD (each disk is exported as is) on a
3ware 9550SX controler.
But as said earlier, for a disk that has a partition in a RAID
an another partition outside of the RAID, the performance are weak
with the RAID and good without it. The underlying hardward hence
seems fine.

At what offset does the partition without RAID start ?

[Emmanuel Dreyfus]

Post by Manuel Bouyer
At what offset does the partition without RAID start ?

I think I found something: I have RAID sets that have decent performances.
The goods and the bads are on different control:

The goods:
raid2
ld13 at twa1 unit 4: 232 GB
ld9 at twa1 unit 0: 232 GB
raid3
ld11 at twa1 unit 2: 149 GB
ld10 at twa1 unit 1: 149 GB
raid4
ld12 at twa1 unit 3: 232 GB
ld14 at twa1 unit 5: 232 GB
raid5
ld15 at twa1 unit 6: 465 GB
ld16 at twa1 unit 7: 465 GB


The bads:
raid0
ld0 at twa0 unit 0: 465 GB
ld3 at twa0 unit 3: 232 GB
raid1
ld1 at twa0 unit 1: 465 GB
ld2 at twa0 unit 2: 232 GB
raid6
ld4 at twa0 unit 4: 931 GB
ld5 at twa0 unit 5: 931 GB
raid7
ld6 at twa0 unit 6: 931 GB
ld7 at twa0 unit 7: 931 GB

But the difference is not obvious:
twa0 at pci2 dev 7 function 0: 3ware 9550SX series (rev. 0x00)
twa0: interrupting at ioapic1 pin 2
twa0: 12 ports, Firmware FE9X 3.08.00.016, BIOS BE9X 3.08.00.004
twa0: Monitor BL9X 3.01.00.006, PCB Rev 032 , Achip 1.70 , Pchip 1.60

twa1 at pci2 dev 8 function 0: 3ware 9550SX series (rev. 0x00)
twa1: interrupting at ioapic1 pin 3
twa1: 12 ports, Firmware FE9X 3.08.00.016, BIOS BE9X 3.08.00.004
twa1: Monitor BL9X 3.01.00.006, PCB Rev 032 , Achip 1.70 , Pchip 1.60

[Greg Troxel]

Post by Emmanuel Dreyfus

Post by Manuel Bouyer
At what offset does the partition without RAID start ?

I think I found something: I have RAID sets that have decent performances.

Try reading from the underlying components of the RAID sets directly.
You can do this when the RAID is configured. If that it is fast and
the rraidN device is slow, that's one thing, but I haven't so far seen
evidence that the issue isn't with the underlying disks.

[Emmanuel Dreyfus]

Post by Greg Troxel
Try reading from the underlying components of the RAID sets directly.
You can do this when the RAID is configured. If that it is fast and
the rraidN device is slow, that's one thing, but I haven't so far seen
evidence that the issue isn't with the underlying disks.

Read performance is fine everywhere, the problem was only for writes.
But I found it, it was a disk cache problem.

[Manuel Bouyer]

Post by Emmanuel Dreyfus
[...]
twa0 at pci2 dev 7 function 0: 3ware 9550SX series (rev. 0x00)
twa0: interrupting at ioapic1 pin 2
twa0: 12 ports, Firmware FE9X 3.08.00.016, BIOS BE9X 3.08.00.004
twa0: Monitor BL9X 3.01.00.006, PCB Rev 032 , Achip 1.70 , Pchip 1.60
twa1 at pci2 dev 8 function 0: 3ware 9550SX series (rev. 0x00)
twa1: interrupting at ioapic1 pin 3
twa1: 12 ports, Firmware FE9X 3.08.00.016, BIOS BE9X 3.08.00.004
twa1: Monitor BL9X 3.01.00.006, PCB Rev 032 , Achip 1.70 , Pchip 1.60

battery backup status, maybe ?
The driver will eventually print informations about this, for example:
twa1: AEN 0x0055: INFO: Battery charging started:
twa1: AEN 0x0056: INFO: Battery charging completed:

tw_cli should also be able to print it, if you have it.

[Emmanuel Dreyfus]

Post by Manuel Bouyer

Post by Emmanuel Dreyfus
raid1 uses ld1a and ld2a
Both ld1a and ld2a have a NetBSD RAID partition starting at sector 63
(very usual stuff).

For large disks this can be a problem. Try to make is start at 2048.

Is 500 GB a large disk?

Post by Manuel Bouyer
At what offset does the partition without RAID start ?

At 488259356. It does not even seems to be aligned.

# size offset fstype [fsize bsize cpg/sgs]
a: 488259293 63 RAID
c: 976541633 63 unused 0 0
d: 976541696 0 unused 0 0
e: 488188340 488259356 4.2BSD 2048 16384

[Manuel Bouyer]

Post by Emmanuel Dreyfus

Post by Manuel Bouyer

Post by Emmanuel Dreyfus
raid1 uses ld1a and ld2a
Both ld1a and ld2a have a NetBSD RAID partition starting at sector 63
(very usual stuff).

For large disks this can be a problem. Try to make is start at 2048.

Is 500 GB a large disk?

I don't know if there has been 500G with 4k sectors.
I think 1T disks with 4K sectors do exists.

Post by Emmanuel Dreyfus

Post by Manuel Bouyer
At what offset does the partition without RAID start ?

At 488259356. It does not even seems to be aligned.

Indeed it's not

[Emmanuel Dreyfus]

Post by Emmanuel Dreyfus
Right, this is not a NFS performance problem, but a RAIDframe performance
problem.

I found the problem: disk cache not enabled on one of the RAID
controllers. Now I have:

On a non RAID partition:
# time dd if=/dev/zero of=/var/crash/z bs=1024k count=100
0.98s real 0.00s user 0.34s system

On a RAID partition:
# time dd if=/dev/zero of=/home5/z bs=1024k count=100
2.53s real 0.00s user 0.36s system


Are the numbers reasonable?

[Manuel Bouyer]

Post by Emmanuel Dreyfus

Post by Emmanuel Dreyfus
Right, this is not a NFS performance problem, but a RAIDframe performance
problem.

I found the problem: disk cache not enabled on one of the RAID
# time dd if=/dev/zero of=/var/crash/z bs=1024k count=100
0.98s real 0.00s user 0.34s system
# time dd if=/dev/zero of=/home5/z bs=1024k count=100
2.53s real 0.00s user 0.36s system
Are the numbers reasonable?

There's still a 2.5x factor, but if it's not the same disk
it can happen.
I can also happen depending on the place of the partition on the disk
(the start of the disk can be faster than the end of the disk).

[Dan LaBell]

Post by Emmanuel Dreyfus
Hello
I have a NFS setup with both NetBSD 6 client and server over a
gigabit network. Theperformance seems week, even whle client,
server and network are almost idle.
The test: time dd if=/dev/zero of=test bs=1024k count=100
5.31s real 0.00s user 0.39s system
9.82s real 0.00s user 0.12s system
The overhead looks huge. This is a UDP mount with a 1500 bytes MTU,
ping is at 0.8 ms from a virtualized client, ang 0.28 ms for a
physical one.
Are the numbers reasonable? Should I consider a 59% NFS overhead as
acceptable, or are there some parameters to tweak?
--
Emmanuel Dreyfus

Don't know what to call it, maybe a block-buffer-interleave issue ,
don't know if it is this ... just throwing it out there ...

BUT, what if that's a big block size (or buffer ) that NFS will
NEVER ever call read with buffers (or blocks that size) ???

Are you using a berkley flavor data-dump that will share with you
when the blocksize is bigger than a buffer dd can pass to read(2) or
write(2) ?
by outputting 0+31459 etc. I believe GNU and also Solaris decided it
(mid '90's) to 'mock-it-up' so the user doesn't get confused...

maybe make the file in /tmp on client once and just 'time cp /tmp/
testfile /nfs/etc/.'
and let the system pick the blocks and buffer sizes naturally.

[Emmanuel Dreyfus]

maybe make the file in /tmp on client once and just 'time cp /tmp/testfile
/nfs/etc/.'
and let the system pick the blocks and buffer sizes naturally.

The result is slower, perhaps because I have to read data from the input
file?