On 27-31 October 2023, Dr. Rubin presented two posters at the 86th Annual Meeting of the Association for Information Science and Technology (ASIS&T 2023): “Making a Difference: Translating Information Research into Practice, Policy, and Action” in London West, London, UK. These short works were co-authored by six doctoral students at the LiT.RL Lab and Dr. Rubin. Both posters offer preliminary results from the Lab’s most recent small project entitled “Assessments of Text-Generating AI (2023)”.

If you are current student at FIMS, you should be able to walk the hallways of the 4th floor FNB Building and find both posters on display.

This research was funded by the FRDF Research Grant at FIMS. Two mini-studies were conducted in response to November 2023 release of ChatGPT and the consequent Microsoft Bing AI Chatbot and Google Bard releases. These text-generating AI systems show increased promise of their capabilities to mimic natural language interactions with computer users and produce plausible coherent narratives in response to prompts. The pace of their adoption by the public is unprecedented. We have established that media coverage steeply increased between December 2022 and March 2023 (Delellis et al., 2023), and has greatly superseding the 2016 media interest around the controversial Microsoft’s Twitter chatbot, Tay. These systems raise wide-ranging concerns about their abilities to pretend, fudge, and produce skillful BS. They step into unexpected roles by producing plagiarized and inaccurate content, and misinform their users, while acting like a capable human news writer, reporter, or any other content producer. Barely distinguishable from human-written narratives, AI-generated content on social media can further undermine the foundations of our educational, political, and communication systems, among others. Chatbots are also on path to destabilize traditional search optimization practices. By late spring 2023, we found Bing Chat, an AI-integrated search engine, showed significantly fewer search results to its users on their first page, as compared to a non-enhanced DuckDuckGo (Kelly et al., 2023). Further project details can be found on my Lab’s website on a specially dedicated webpage.

Here are some further details on these two studies including their abstracts and suggested citations:

Delellis, Nicole, S., Chen, Yimin; Cornwell, Sarah, E., Kelly, Dominique; Mayhew, Alex; Onaolapo, Sodiq; and Rubin, Victoria, L. (2023). “ChatGPT Media Coverage Metrics: Initial Examination.” In the Proceedings of the the 86th Annual Meeting of the Association for Information Science & Technology | Oct. 29 – 31, 2023 | London, United Kingdom.

ABSTRACT: This paper presents an overview of coverage of OpenAI’s ChatGPT in media outlets from November 2022-March 2023, a comparison to previous media coverage of the chatbot Tay across the same outlets, and a count of ChatGPT media articles pertaining to government legislation and regulation. The New York Times, Wired, Gizmodo, The Globe and Mail, and The Guardian were searched for coverage. Across all five outlets there is an uptick in media coverage surrounding ChatGPT, with total numbers of included articles per month being 0 in November, 39 in December, 68 in January, 104 in February, and 143 in March. Findings exemplify the trend of increased coverage of ChatGPT in media public discourse, which contrasts with previous smaller media coverage of Tay. Examination of headlines and subheadings of included articles reveals minimal coverage (5.7%) dedicated to government legislation of ChatGPT. Future research will evaluate what is being said about ChatGPT within these media outlets.  KEYWORDS: ChatGPT; Media Coverage; Tay; Legislation

Kelly, Dominique; Chen, Yimin; Cornwell, Sarah, E.; Delellis, Nicole; Mayhew, Alex; Onaolapo, Sodiq; and Rubin, Victoria, L. (2023). “Bing Chat: The Future of Search Engines?” In the Proceedings of the the 86th Annual Meeting of the Association for Information Science & Technology | Oct. 29 – 31, 2023 | London, United Kingdom.

ABSTRACT: Introduced by Microsoft in February 2023, Bing Chat is a feature of the Bing search engine that integrates an OpenAI large language model (LLM) customised for search (Mehdi, 2023a). This poster compares the outputs of Bing Chat and a standard existing search engine (DuckDuckGo) in response to identical keyword queries and corresponding natural language (NL) questions. Specifically, we examined: (1) the length of Bing Chat’s responses and DuckDuckGo’s first page of search results, by number of website links; and, (2) the length of Bing Chat’s textual summaries, by number of website links. We found that, on average, significantly fewer websites were linked to in Bing Chat’s responses compared to DuckDuckGo’s search results. Our findings have important implications for website operators, who may receive less traffic and ad revenue if LLM-enabled search engines are widely adopted in the future. Human-Computer Interaction (HCI) will inevitably face the need for more research on human information behaviours adaptations in response to the changing search paradigm.
KEYWORDS: Search Engines; Bing Chat; DuckDuckGo; Large Language Models; ChatGPT