Wednesday, December 23, 2009

Qtree SnapMirror update states that file system is full

Qtree SnapMirror update states that file system is full

View Environment section

Symptoms
Qtree SnapMirror update states that file system is full
Snapmirror fails with error: [snapmirror.dst.waflErr:error]: SnapMirror destination transfer from filer:/vol/srcvol/srcqtree to /vol/dstvol/dstqtree : qtree snapmirror destination write failed: No space left on device.

Cause of this problem

For qtree snapmirror (QSM), it is possible that the destination filer's volume containing the destination qtree for a snapmirror relationship will require more space than the volume on the source filer.  This occurs for two reasons:

  1. During a qtree snapmirror update, changes must be replicated to the destination before data can be removed.  Thus, at least 5% free space should be available per QSM relationship to allow for the temporary space needed during the transfer.

  2. Volume-level snapshots independent of the QSM base snapshot  (such as the nightly snapshots) can retain data in the QSM qtrees. Since the volume snapshots are unique to the destination filer, they may contain data that was already deleted from the source filer.

If the destination qtree has enough space to hold the data in the source qtree, but it does not have enough space to contain the source qtree plus the snapshot delta, the QSM update will fail with the following error:
[snapmirror.dst.waflErr:error]: SnapMirror destination transfer from srcfiler:/vol/srcvol/srcqtree to /vol/dstvol/dstqtree : qtree snapmirror destination write failed: No space left on device.


Solution

Increase the size of the destination volume so that it can hold the sum of the following:

  • The amount of data stored in the source qtree
  • The amount of data stored in snapshots
  • 5% free space

The following scenario is an example of how the destination volume for a QSM relationship can become full even though it is the same size as the source filer's volume. A QSM relationship is set up from source srcfiler:/vol/srcvol/srcqtree to destination dstfiler:/vol/dstvol/dstqtree. The srcvol on srcfiler is 100GB in size and contains a qtree using 20 GB.  A "df -g" on this filer shows:

srcfiler> df -g
Filesystem total used avail capacity
/vol/srcvol/ 80GB 20GB 60GB 25%  
/vol/srcvol/.snapshot 20GB  0GB 20GB  0%   

Volume snapshots are disabled on the source volume /vol/srcvol. The dstvol on dstfiler is also 100GB in size and contains the QSM destination qtree, which holds the same 20GB of data as the source qtree (srcfiler:/vol/srcvol/srcqtree).  A "df -g" on dstfiler shows:

dstfiler> df -g
Filesystem total used avail capacity
/vol/dstvol/ 80GB 20GB 60GB 25% 
/vol/dstvol/.snapshot 20GB  0GB 20GB  0%
Volume snapshots are enabled on the destination volume /vol/dstvol:

dstfiler> snap sched
Volume dstvol: 0 2 6@8,12,16,20

The nightly snapshot has been taken on the destination volume.  This snapshot contains the 20GB of data in /vol/dstvol/dstqtree that was replicated over by QSM.
dstfiler> snap list dstvol
Volume dstvol
working...

%/used %/   total       date         name
---------- ---------- ------------ --------
25% ( 25%) 10% ( 10%) Aug 20 16:00 nightly.0
On the source filer, 20 GB of files are added, and 10 GB of files are deleted.  The srcvol now contains 30GB of data:

srcfiler> df -g
Filesystem total used avail capacity
/vol/srcvol/ 80GB 30GB 50GB 38%
/vol/srcvol/.snapshot 20GB  0GB 20GB  0% 

A QSM update occurs, and the changes are replicated to the destination. QSM sends delete information for the files which have been deleted, and complete data for the newly created files.  Thus, the data transferred is equal to 20 GB of new data + (4KB * number of deleted files).  Once the transfer completes, the destination filer now has the following space used:

dstfiler> df -g
Filesystem total used avail capacity
/vol/dstvol/ 80GB 30GB 50GB 38% 
/vol/dstvol/.snapshot 20GB 10GB 20GB 50%

Notice that there is space used in the destination filer's snap reserve.  This space is held in the nightly snapshot:

dstfiler> snap list dstvol
Volume dstvol
working...

%/used %/     total     date        name
---------- ---------- ------------ --------
25% ( 25%) 10% ( 10%) Aug 20 16:00 nightly.0

Thus, because the destination volume has snapshots enabled, it can use more space that the source volume. Therefore, it is important to consider the snapshot retention period and snapshot delta when sizing the destination volume for QSM relationships.

Last Updated:  24 AUG 2006

Environment
EnvironmentBookmark
Data ONTAP
All NetApp filer
NearStore
SnapMirror

If you would like more help, please try the new NOW Support Communities where registered customers, partners, and NetApp technical experts discuss technical questions and issues.

No comments:

Post a Comment