ngshare is ignoring c.CourseDirectory.ignore?

[JBRhoads]

We're seeing an issue where students submissions are including the .ipynb_checkpoints folder. But when the instructor collects the submissions, the .ipynb_checkpoints folder isn't downloaded even though nbgrader is still expecting it. This is a fatal error and the collection process won't continue even to the next student. Essentially, there's a mismatch where one side is ignoring but the other isn't. Currently, the only solution is for the students to manually delete the .ipynb_checkpoints folder before submitting.

[CollectApp | INFO] Decoding: /home/jovyan/submitted/user1/Major Final/Untitled.ipynb
[CollectApp | INFO] Decoding: /home/jovyan/submitted/user1/Major Final/thing.py
[CollectApp | INFO] Decoding: /home/jovyan/submitted/user1/Major Final/thing/thing.py
[CollectApp | INFO] Decoding: /home/jovyan/submitted/user1/Major Final/.test/thing.txt
[CollectApp | INFO] Decoding: /home/jovyan/submitted/user1/Major Final/.ipynb_checkpoints/thing.txt
Traceback (most recent call last):
File "/usr/local/bin/nbgrader", line 8, in
sys.exit(main())
File "/home/jovyan/.local/lib/python3.9/site-packages/nbgrader/apps/nbgraderapp.py", line 330, in main
NbGraderApp.launch_instance()
File "/usr/local/lib/python3.9/site-packages/jupyter_core/application.py", line 269, in launch_instance
return super().launch_instance(argv=argv, **kwargs)
File "/home/jovyan/.local/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
app.start()
File "/home/jovyan/.local/lib/python3.9/site-packages/nbgrader/apps/nbgraderapp.py", line 322, in start
super(NbGraderApp, self).start()
File "/home/jovyan/.local/lib/python3.9/site-packages/nbgrader/apps/baseapp.py", line 362, in start
super(NbGrader, self).start()
File "/usr/local/lib/python3.9/site-packages/jupyter_core/application.py", line 258, in start
self.subapp.start()
File "/home/jovyan/.local/lib/python3.9/site-packages/nbgrader/apps/collectapp.py", line 104, in start
collect.start()
File "/usr/local/lib/python3.9/site-packages/ngshare_exchange/exchange.py", line 235, in start
return super(Exchange, self).start()
File "/home/jovyan/.local/lib/python3.9/site-packages/nbgrader/exchange/abc/exchange.py", line 84, in start
self.copy_files()
File "/usr/local/lib/python3.9/site-packages/ngshare_exchange/collect.py", line 148, in copy_files
self.do_copy(submission['files'], dest_path)
File "/usr/local/lib/python3.9/site-packages/ngshare_exchange/collect.py", line 167, in do_copy
self.decode_dir(src, dest)
File "/usr/local/lib/python3.9/site-packages/ngshare_exchange/exchange.py", line 188, in decode_dir
with open(dest_path, 'wb') as d:
FileNotFoundError: [Errno 2] No such file or directory: '/home/jovyan/submitted/user1/Major Final/.ipynb_checkpoints/thing.txt'

I've confirmed that this file exists on within the exchange, but I guess it's refusing to copy the file even though nbgrader is expecting it.

The default variable in nbgrader_config.py (which I've also manually set just to be certain):
c.CourseDirectory.ignore = ['.ipynb_checkpoints', '*.pyc', '__pycache__', 'feedback']

[asvattha]

I am also having the same error. Did you find any workaround for this? It's common for students to forget to delete the .ipynb_checkpoints before submitting.

[JBRhoads]

Yes, here's the workaround I'm using. First, find where your ngshare.db is. Also, you have to install sqlite3.

1. To view the submissions which contain .ipynb_checkpoints files:
   sqlite3 ngshare.db 'select * from submission_files_assoc_table join files on files._id=submission_files_assoc_table.right_id where files.filename like "%.ipynb_checkpoints%";' '.exit'

2. I suggest backing up your ngshare.db file, just in case.

3. Then to delete the associations:
   sqlite3 ngshare.db 'delete from submission_files_assoc_table where ROWID IN (select submission_files_assoc_table.ROWID from submission_files_assoc_table join files on files._id=submission_files_assoc_table.right_id where files.filename like "%.ipynb_checkpoints%");' '.exit'

4. Run the first command again to see that the files are now unassociated.

To make this easier for me, I'm just running the delete command every night via a CronJob deployment in k8s.

[asvattha]

@JBRhoads

1. How to find the ngshare.db? is it same as the gradebook.db of nbgrader?
2. To install sqlite3; you mean to install it inside the pod (container) or on k8s?
3. Your code in Step 1 and Step 3 commands are exactly same, how the latter is deleting the associations?

[JBRhoads]

Oops, I fixed step 3.

No, ngshare.db will be in the ngshare server pod. Yeah, you can install sqlite3 there (I haven't tried this) or you can on whichever system that can also access that same persistent volume that's being used by ngshare.

[asvattha]

@JBRhoads
Thank you very much for such a quick response. It's working as per your suggestion. :)

[mike-pisman]

Yes, here's the workaround I'm using. First, find where your ngshare.db is. Also, you have to install sqlite3.

1. To view the submissions which contain .ipynb_checkpoints files:
   `sqlite3 ngshare.db 'select * from submission_files_assoc_table join files on files._id=submission_files_assoc_table.right_id where files.filename like "%.ipynb_checkpoints%";' '.exit'`

2. I suggest backing up your ngshare.db file, just in case.

3. Then to delete the associations:
   `sqlite3 ngshare.db 'delete from submission_files_assoc_table where ROWID IN (select submission_files_assoc_table.ROWID from submission_files_assoc_table join files on files._id=submission_files_assoc_table.right_id where files.filename like "%.ipynb_checkpoints%");' '.exit'`

4. Run the first command again to see that the files are now unassociated.

To make this easier for me, I'm just running the delete command every night via a CronJob deployment in k8s.

Thank you sharing, this worked. Can you please also share the manifest for the cron job?