Twitter Datasets and Derivative data

Last updated on 7 min read

#healthcanada #NACI #fordnation #medicalfreedom #covid19 #covid19vaccines #protectourfamilies #protectyourchildren #holdtheline tweets

2,661,117 tweet ids for #healthcanada #NACI #fordnation #medicalfreedom #covid19 #covid19vaccines #protectourfamilies #protectyourchildren #holdtheline tweets, collected with Documenting the Now's twarc. Tweets can be “rehydrated” with Documenting the Now’s twarc, or Hydrator.

twarc hydrate tweet-ids.txt > tweets.jsonl

ID files are available for all hashtags or some individual hashtags:

  • covid19-ids.txt
  • covid19vaccines-ids.txt
  • fordnation-ids.txt
  • healthcanada-ids.txt
  • healthcanada-NACI-fordnation-medicalfreedom-covid19-covid19vaccines-protectourfamilies-protectyourchildren-holdtheline-ids.txt
  • holdtheline-ids.txt
  • medicalfreedom-ids.txt
  • NACI-ids.txt
  • protectyourchildren-ids.txt

Tweets were collected via the Standard Search API on:

  • November 18, 2021
  • November 21, 2021
  • November 26, 2021
  • December 1, 2021

#elxn44 tweets (44th Canadian Federal Election)

2,075,645 tweet ids for #elxn44 tweets, collected with Documenting the Now's twarc. Tweets can be “rehydrated” with Documenting the Now’s twarc, or Hydrator.

twarc hydrate elxn44-tweet-ids.txt > elxn44.jsonl.

Tweets were collected via the Standard Search API on a cron job every five days from July 28, 2021 - November 05, 2021.

Wet’suwet’en tweet ids

425,227 tweet ids for Wet'suwet'en tweets, collected with Documenting the Now's twarc. Tweets can be “rehydrated” with Documenting the Now’s twarc, or Hydrator.

twarc hydrate wetsuweten-20210115-ids.txt > wetsuweten.jsonl

Tweets were collected via the Standard Search API on a cron job every five days beginning on February 18, 2020. Collection is ongoing.

The account that was used to collect these Tweets failed to collect Tweets for the period from Sun Jul 26 02:00:21 +0000 2020 through Fri Aug 07 20:05:54 +0000 2020.

Tyendinaga tweet ids

80,264 tweet ids for Tyendinaga tweets, collected with Documenting the Now's twarc. Tweets can be “rehydrated” with Documenting the Now’s twarc, or Hydrator.

twarc hydrate tyendinaga-20210115-ids.txt > tyendinaga.jsonl

Tweets were collected via the Standard Search API on a cron job every five days beginning on February 24, 2020. Collection is ongoing.

The account that was used to collect these Tweets failed to collect Tweets for the period from Sun Jul 26 03:46:41 +0000 2020 through Fri Aug 07 20:33:27 +0000 2020.

#elxn43 tweets (43rd Canadian Federal Election)

2,944,525 tweet ids for #elxn43 tweets, collected with Documenting the Now's twarc. Tweets can be “rehydrated” with Documenting the Now’s twarc, or Hydrator.

twarc hydrate elxn43-ids.txt > elxn43.jsonl

Tweets were collected via the Standard Search API on a cron job every five days from September 9, 2019 - November 23, 2019.

Tweets to Donald Trump (@realDonaldTrump)

362,464,578 tweet ids for tweets directed at Donald Trump (@realDonaldTrump), collected with Documenting the Now's twarc. Tweets can be “rehydrated” with Documenting the Now’s twarc, or Hydrator.

twarc hydrate to_realdonaldtrump_20210120_ids.txt > to_realdonaldtrump_20210120.jsonl.

Collection notes:

  • Tweets from May 7, 2017 - October 16, 2018 of the dataset used a combination of the Filter (Streaming) API and Search API.
  • The Filter API failed on June 21, 2017.
  • From June 23, 2017 forward only the Search API was used to collect.
  • Collection was done every 5 days on a cron job, and periodically deduplicated.
  • There is a data gap from Tue Jul 28 13:53:50 +0000 2020 through Thu Aug 06 09:36:23 +0000 2020 due to a collection error.

This dataset also includes a number of derivative csv files from the original jsonl collected. This includes:

User csv:

jq -r '[.id_str, .created_at, .user.screen_name, .retweeted_status != null] | @csv' to_realdonaldtrump_20190130.jsonl > to_realdonaldtrump_20190130_users.jsonl

#JeffSessions tweets

2,278,757 tweet ids for #JeffSessions collected with Documenting the Now's twarc. Tweets can be “rehydrated” with Documenting the Now’s twarc, or Hydrator.

twarc hydrate to_realdonaldtrump_ids.txt > to_donaltrump.jsonl.

#paradisepapers tweets November 3-25, 2017

1,797,260 tweet ids for #paradisepapers collected with Documenting the Now’s twarc from November 5-26, 2017. Tweets can be “rehydrated” with Documenting the Now’s twarc: twarc.py hydrate paradisepapers_ids.txt > paradisepapers.json. Or with Documenting the Now’s Hydrator.

#climatemarch tweets April 19-May 3, 2017

681,668 tweet ids for #climate collected with Documenting the Now’s twarc from January 22-26, 2017. Tweets can be “rehydrated” with Documenting the Now’s twarc: twarc.py hydrate climatemarch_tweet_ids.txt > climatemarch.json.

#MarchForScience tweets April 12-26, 2017

1,276,220 tweet ids for #MarchForScience collected with Documenting the Now’s twarc from January 22-26, 2017. Tweets can be “rehydrated” with Documenting the Now’s twarc: twarc.py hydrate MarchForScience_tweet-ids.txt > MarchForScience.json.

#WomensMarch tweets January 12-28, 2017

14,478,518 tweet ids for #WomensMarch collected with Documenting the Now’s twarc from January 21-28, 2017. Tweets can be “rehydrated” with Documenting the Now’s twarc: twarc.py –hydrate WomensMarch_tweet_ids.txt > WomensMarch.json. Also included are the logs files for the Filter API and Search API queries. The Filter API query captures the cumulative number of dropped tweets.

The fall of Aleppo tweets; Aleppo 2016-12-13 through 2016-12-29

8,595,589 tweet ids for aleppo tweets captured during the fall of Aleppo in December 2016. Tweets can be “rehydrated” with Documenting the Now’s twarc: twarc.py –hydrate aleppo_tweet_ids.txt > aleppo.json

Tweet ids for final Tragically Hip concert

228,086 tweet ids for “TheHip, hipinkingston” captured during the Tragically Hip’s final concert in Kingston, Ontario in August 2016. Tweets can be “rehydrated” with Documenting the Now’s twarc: twarc.py –hydrate th_final_concert_kingston_tweet_ids.txt > th_final_concert_kingston.json.

#YMMfire tweets

Tweet ids for #YMMfire tweets. Tweets can be “hydrated” with Ed Summers’ twarc: twarc.py –hydrate ymmfire-ids.txt > ymmfire-tweets.json. Hydrating will recreate the original tweet(s) in json format, provided the content is still available on Twitter.

#jcdl2016 tweets

Tweet ids for #jcdl2016 tweets. Tweets can be “hydrated” with Ed Summers’ twarc: twarc.py –hydrate jcdl2016-tweet-ids.txt > jcdl2016-tweets.json. Hydrating will recreate the original tweet(s) in json format, provided the content is still available on Twitter.

#thechalkening tweets

Tweet ids for #thechalkening tweets. Tweets can be “hydrated” with Ed Summers’ twarc: twarc.py –hydrate thechalkening-ids-20160412.txt > thechalkening-20160412-tweets.json. Hydrating will recreate the original tweet(s) in json format, provided the content is still available on Twitter.

#panamapapers tweets

Tweet ids for #panamapapers tweets. Tweets can be “hydrated” with Ed Summers’ twarc: twarc.py –hydrate panamapapers-ids-20160413.txt > panamapapers-20160413-tweets.json. Hydrating will recreate the original tweet(s) in json format, provided the content is still available on Twitter.

#NDP2016 tweets

#MakeDonaldDrumpfAgain tweets

Derivative data for #MakeDonaldDrumpfAgain tweets. Tweets can be “hydrated” with Ed Summers’ twarc: twarc.py –hydrate MakeDonaldDrumpfAgain-tweet-ids.txt > MakeDonaldDrumpfAgain.json. Hydrating will recreate the original tweet(s) in json format, provided the content is still available on Twitter. This dataset is the combination of hydrated http://hdl.handle.net/10864/11310 tweet ids, and htttp://hdl.handle.net/10864/11270.

#paris #Bataclan #parisattacks #porteouverte tweets

Tweet ids for #paris #Bataclan #parisattacks #porteouverte tweets. Tweets can be “hydrated” with Ed Summers’ twarc: twarc.py –hydrate paris-tweet-ids.txt > paris-tweets.json. Hydrating will recreate the original tweet(s) in json format, provided the content is still available on Twitter.

#elxn42 tweets (42nd Canadian Federal Election)

Tweet ids for #elxn42 tweets. Tweets can be “hydrated” with Ed Summers’ twarc: twarc.py –hydrate elxn42-tweet-ids.txt > elxn42-tweets.json. Hydrating will recreate the original tweet(s) in json format, provided the content is still available on Twitter. This dataset is the combination of hydrated http://hdl.handle.net/10864/11310 tweet ids, and htttp://hdl.handle.net/10864/11270.

#JeSuisCharlie, #JeSuisAhmed, #JeSuisJuif, #CharlieHebdo tweets

Tweet ids for #JeSuisCharlie, #JeSuisAhmed, #JeSuisJuif, #CharlieHebdo tweets. Tweets can be “rehydrated” with Ed Summers’ twarc: twarc.py –hydrate %23JeSuisCharlie-ids-20150112.txt > %23JeSuisCharlie-tweets-20150112.json

twitter datasets twarc
Avatar
Nick Ruest
Associate Librarian

Related