Skip to content

Latest commit

 

History

History
83 lines (65 loc) · 4.22 KB

README.md

File metadata and controls

83 lines (65 loc) · 4.22 KB

Comparing Inferential Strategies of Humans and LLMs in Deductive Reasoning

automated reasoning

Generated by DALL·E 3


Deductive reasoning plays a pivotal role in the formulation of sound and cohesive arguments. It allows individuals to draw conclusions that logically follow, given the truth value of the information provided. Recent progress in the domain of large language models (LLMs) has showcased their capability in executing deductive reasoning tasks. Nonetheless, a significant portion of research primarily assesses the accuracy of LLMs in solving such tasks, often overlooking a deeper analysis of their reasoning behavior. In this project, we draw upon principles from cognitive psychology to examine inferential strategies employed by LLMs, through a detailed evaluation of their responses to propositional logic problems.

This repository contains code to evaluate the reasoning behavior of LLMs in problems of propositional logic. In particular, we assess inferential strategies employed by LLMs on 12 tasks suggested by Van der Henst et al. (2002). The problem formulations for each task can be found in our respective HuggingFace data repository.

Table of Contents

Setup

All code was developed and tested on Ubuntu 22.04 with Python 3.11.6.

To run the current code, we recommend to use Poetry:

poetry install                          # Install dependencies
poetry shell                            # Activate virtual environment
# Work for a while
deactivate

Instructions

To run the code, you can use the following command:

python run.py --model <model-name> --num-samples <num-samples>

where <model-name> is the name of the LLM to be used and <num-samples> is the number of random seeds across which the model should be evaluated. In this project, we used the following models:

Note that in order to use a new model, you need to add a configuration file in this folder.

Model Responses and Human Annotations

We publish all model answers and human annotations in our respective HuggingFace data repository: inferential-strategies.

License

CC BY 4.0

This work is licensed under a Creative Commons Attribution 4.0 International License.

Citation

If you find this work helpful, please our paper as:

@inproceedings{mondorf-plank-2024-comparing,
    title = "Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning",
    author = "Mondorf, Philipp  and Plank, Barbara",
    editor = "Ku, Lun-Wei  and Martins, Andre  and Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.508",
    doi = "10.18653/v1/2024.acl-long.508",
    pages = "9370--9402"
}