Displaying item access status in search results in DSpace 7

[pilasou]

In DSpace 7, a user can only know if an item is open access or not by opening the item record.

User must be able to know quickly in the UI if the main file of an item is Open Access or not. One first step would be to display the access status on search results.

The item metadata dc.date available cannot be rely upon to determine access status as it is not consistently used by DSpace to display the Open Access date of the item (see https://groups.google.com/g/dspace-tech/c/k7qCSJVhRrU and DSpace/DSpace#6583).

In DSpace, access restrictions are set on bitstream. As an item can have one to many bitstream associated to it with different access restrictions, access status on the item should be based on the main file of the item (i.e. full text of an article, thesis, etc.). Hypothesis is that the first file is the main file in DSpace original bundle.

4 access status can be displayed based on the restriction value used in DSPace 7 workflow:

Statut (anglais)	Description
Open Access	Value Open access  in DSpace Workflow is selected for the file
Embargo	Value embargo in DSpace Workflow is selected for the file
Restricted	Value Administrator in DSpace Workflow is selected for the file
Metadata only	No bitstream associated

The following process can be used to determine the access status:

1. Look at all READ policies of the primary/first bitstream,
2. If one policy is bond to the Anonymous group and if the current date is between the start date and the end date of this policy, then item status is OpenAccess.
3. Otherwise: if one policy for the Anonymous group has a start date inferior to the specific predefined date set in the configuration (for infinite embargo), then, it is an embargo item.
4. If the start date is superior or equal the date specified in the configuration, then, the item is restricted.
5. If no policy for the Anonymous group is set, the item is considered Restricted.
6. Finally, if the item has no bitstream, the Item status will be Metadata only.

A couple of parameters would be added to the configuration. First off, the access status display in search results can be activated or deactivated as a configuration parameter in DSpace. Next, the forever date would also be added to the configuration with a year, a month and a day.

Access status will be displayed as tags as illustrated below:

[nibou230]

I'm in charge of this feature development at Université Laval. As requested by @tdonohue, here's a quick overview of how I was planning to do it.

First of all, this feature doesn't require any modification to the database or Solr. I was hoping to fetch everything I need from the UI (dspace-angular) without doing any modification on the server, but I'm not convinced it's really possible after my investigations. For now, I can only fetch the policies if the logged in user (let’s say the administrator) has access to them. With those policies and the associated dates in hand, I can calculate the access status and show it as a badge. Should or must I create a method on the server to calculate this status and return it to the UI instead?

I'm planning to add a new component named AccessStatusBadgeComponent. This one would be similar to TypeBadgeComponent. In addition to this new component, we were planning to add a mini configuration section in the YAML files to enable this feature. An example is visible in the image below.

I would greatly appreciate a feedback to see if I'm missing something and if I'm on the right track at this point.

[tdonohue]

@nibou230 : My immediate response is that you may hit major performance problems if you attempted to do most of the "logic" (behind these badges) in the Angular UI codebase. I'd worry that, at best, you'd need to make several additional queries to the REST API per Item....which could be very slow on any search/browse page that is displaying more than 10 items at once. (While the default is 10 items per page, it is configurable to display more.)

I suspect it'd have much better performance if the logic could be done on the backend. We might consider indexing this info into Solr so that it can be returned as part of the Item response (e.g. whenever the Item is returned, it has an "accessStatus"). However, then the challenge would be in knowing when to trigger a re-index of the Item (i.e. how to know when the status has changed). Unfortunately, embargoes could be the hardest to deal with, as when the embargo expires, there's no modification of the Item, so there's no way to know it needs a new "accessStatus" without checking if the embargo has expired yet.

We also need to keep in mind that some sites choose to add Embargoes or Access Restrictions to the Item instead of the Bitstream(s). So, the "Embargoed" and "Restricted" badges may be more complex than they appear, since the embargo/restriction could be at the Item level or the Bitstream level or potentially both...e.g. Item can be Restricted to a limited group of users, while it's Bitstream is also Embargoed until a date passes (Example use case: an Item may only be available "On Campus", but only after a specific date passes. So, it's temporarily admin-only, but once the embargo passes, it's "On Campus" access.)

Overall, I like the general idea... but, I'm not sure how to easily implement it in DSpace 7. It'd also need to be planned carefully to avoid performance problems.

Per @pilasou 's request via email, I'll add this to our DSpace 7 meeting agenda to see if others have ideas in that meeting.

[pilasou]

@tdonohue Thanks! Our goal is, as a starting point, to display the access status only for the anonymus group. This is the basis of the contribution we want to make. We want to implement it as a configuration in DSpace (people can easily switch on/off). We see some similarities with the Request Copy feature in Dspace (are we mistaken?) Let's discuss all this on Thurday then! :)

[abollini]

We have managed this kind of feature in the past (version 5 & 6) and we are happy to share what learned.

1. calculate the access status for each request, for each item individually, can be very time consuming and lead to performance issue. Moreover, it is quite common for institution to request to index this status so that a facet/index can be provided on it as well; to mitigate that we have stored the result in an item metadata that was recalculated in case of edit/changes
2. there are different ways to "summarize" the access status of an item when multiple files are present. This mean that calculation should be put in a "pluggable" class so that different implementations can be easily offered. The main approaches that we have seen are: look only to the primary bitstream policies (look for the code used to set the google scholar file metatag citation_pdf_url) or try to summarize the policies for all the bitstreams https://github.com/4Science/DSpace/blob/dspace-5_x_x-cris/dspace-api/src/main/java/org/dspace/content/integration/defaultvalues/FulltextPermissionGenerator.java
3. in the current version there is a solr index plugin that can be eventually a base for your development https://github.com/DSpace/DSpace/blob/main/dspace-api/src/main/java/org/dspace/discovery/SolrServiceContentInOriginalBundleFilterPlugin.java

Due to the point 1 above, if you keep this information only in SOLR you should really make changes to the search endpoint to return solr fields directly so that this information can be offered by the REST API to the angular UI without the need to recalculate it. There is some support to specify with SOLR fields are retrieved here https://github.com/DSpace/DSpace/blob/main/dspace-api/src/main/java/org/dspace/discovery/SolrServiceImpl.java#L747-L755 but these fields are not included in the returned REST resource.

Last but not least, indexing such information in SOLR calculating it during indexing can slow down indexing performance, this is why in the past we have "consolidated" the information in an item metadata. An alternative to item metadata could be to create a separate dedicated storage (table) for such information and have the solr plugin to retrieve it from there.
You also need to consider that a script is required to retrieve items that could have changed their access status due to expiration of policies or embargos. The easier way to accomplish that is to index in solr the day when an access status update is needed and run a solr query that will retrieve all the item where such field is included in the range * TO NOW.
Depending on the number of items that could hit access status change in a single day or in the update windows it could be problematic to reindex all of them and could be appropriate to perform SOLR atomic update to just change the access status field. We use a such approach in DSpace-CRIS here https://github.com/4Science/DSpace/blob/dspace-cris-7/dspace-api/src/main/java/org/dspace/discovery/SolrServiceImpl.java#L1572

[nibou230]

@abollini Thanks! I'm happy you took the time to write this message.

It gives many hints at how we can implement this now and in the future for improvements. We will take the time to analyze all those possibilities considering the time we have left on the project.

This will help us a lot going forward!

[tdonohue]

@nibou230 : Since you are working on this, I've assigned this ticket back to you. I've also moved it over into our 7.3 board, just for tracking purposes... however, that doesn't mean it will definitely be in 7.3, it just means it's under consideration if all things go well.

[pnbecker]

Thanks to @tdonohue to make me aware of this issue here.

In DSpace 6 there was a feature that allowed to add visual indicators of different kinds to browse and search results. I know that it was used to mark items as Open Access depending on metadata like dc.rights or dc.rights.uri. As far as I know this is not ported to DSpace 7 yet. I just created an issue to discuess whether we want to port or to remove it from DSpace 7: DSpace/DSpace#8244.

Maybe it is worth to take a look on the solution we had in DSpace 6 JSPUI for this (see the issue linked above for more details). If I remember correctly, the benefit of the solution in DSpace 6 was, that indicators for all kind of topics, not only for permissions and licenses, could be realized.

I'd be grateful if you could take a look and maybe add a comment to DSpace/8244 whether you think the code might be useful for what you try to achieve here.

[pnbecker]

I don't know that JSPUI feature very well, but seeing class names like org.dspace.app.itemmarking.ItemMarkingCollectionStrategy and org.dspace.app.itemmarking.ItemMarkingAvailabilityBitstreamStrategy and taking a look on item-marking.xml suggest that it is easy to extend the ideas about what you want to use visual markers in search and browse results for and to make the feature discussed here a bit more general.

[nibou230]

@pnbecker Yes, I looked at the old code and this could be nice to port this feature in DSpace 7 in the future by converting all the badges to something more generic. Sadly, as you may know, I don't have much time and I won't be able to do and include this work in my future pull request. Maybe it could be discussed furthermore in the new issue.

[selensoftr]

is possible show Embargo Access Date in Bitstream File in DSpace 7.6 UI Angular?

Example DSpace 6 JSPUI

[tdonohue]

@selensoftr : No, that's in another ticket #2413