Goal === Fix the error in DLE logs: ``` 2021/01/25 04:16:40 [ERROR] Failed to check the idleness of clone: failed to get the last session activity: failed to init file selector: failed to read a log directory: open /var/lib/dblab/clones/dblab_clone_6002/data/log: no such file or directory. ``` In `dblab_server`: ``` bash-5.0# ls -la /var/lib/dblab/clones/dblab_clone_6002 total 9 drwxr-xr-x 2 root root 2 Dec 28 12:01 . drwxr-xr-x 276 root root 276 Jan 25 00:00 .. ``` In clone container: ``` ubuntu@db-clones:~/.dblab$ sudo docker exec -it dblab_clone_6002 bash root@99c8113376eb:/# ls -la /var/lib/dblab/clones/dblab_clone_6002/data/log | head total 21546 drwx------ 2 postgres postgres 3092 Jan 25 04:00 . drwx------ 21 postgres postgres 36 Jan 25 04:00 .. -rwx------ 1 postgres postgres 2256 Dec 24 11:17 postgresql-2020-12-24_111738.csv -rwx------ 1 postgres postgres 144 Dec 24 11:17 postgresql-2020-12-24_111738.log -rwx------ 1 postgres postgres 2226 Dec 24 11:17 postgresql-2020-12-24_111751.csv -rwx------ 1 postgres postgres 144 Dec 24 11:17 postgresql-2020-12-24_111751.log -rwx------ 1 postgres postgres 2218 Dec 24 11:18 postgresql-2020-12-24_111759.csv -rwx------ 1 postgres postgres 143 Dec 24 11:18 postgresql-2020-12-24_111759.log -rwx------ 1 postgres postgres 2226 Dec 24 11:18 postgresql-2020-12-24_111813.csv ``` It seems the reason is that the mount points have been removed while snapshots cleaned up ``` 2021/01/03 18:00:00 [ERROR] failed to clean up snapshots automatically: failed to clean up snapshots: failed to clean up snapshots: RunnerError(cmd="zfs list -t snapshot -H -o name -s dblab:datastateat -s creation -r dblab_pool | grep -v clone | head -n -4 | xargs -n1 --no-run-if-empty zfs destroy -R", inerr="exit status 123", stderr="umount: can't unmount /var/lib/dblab/clones/clone_pre_20201224121041 : Resource busy cannot unmount '/var/lib/dblab/clones/clone_pre_20201224121041': umount failed cannot destroy snapshot dblab_pool@snapshot_20201224121041_pre: snapshot is cloned cannot destroy 'dblab_pool/dblab_clone_6000': dataset is busy cannot destroy snapshot dblab_pool/clone_pre_20201226180000@snapshot_20201226175931: snapshot is cloned cannot destroy snapshot dblab_pool@snapshot_20201226180000_pre: snapshot is cloned cannot destroy 'dblab_pool/dblab_clone_6001': dataset is busy cannot destroy snapshot dblab_pool/clone_pre_20201227120000@snapshot_20201227114009: snapshot is cloned cannot destroy snapshot dblab_pool@snapshot_20201227120000_pre: snapshot is cloned cannot destroy 'dblab_pool/dblab_clone_6002': dataset is busy cannot destroy snapshot dblab_pool/clone_pre_20210102180000@snapshot_20210102175434: snapshot is cloned cannot destroy snapshot dblab_pool@snapshot_20210102180000_pre: snapshot is cloned " exit="123") ``` TODO / How to implement === - avoid forced deletion - remove temporary created `_pre` snapshots and clones Acceptance criteria ===