Datasets:
winograd_wsc

Tasks:

Multiple Choice

Sub-tasks: multiple-choice-coreference-resolution

Languages: English

Multilinguality: monolingual

Size Categories: n<1K

Language Creators: expert-generated

Annotations Creators: expert-generated

Source Datasets: original

License: cc-by-4.0

Dataset card Files Files and versions Community

Dataset Viewer

Auto-converted to Parquet

Go to dataset viewer

Viewer

text string	pronoun string	pronoun_loc int32 8 163	quote string	quote_loc int32 8 156	options sequence	label class label 2 classes	source string
"The city councilmen refused the demonstrators a permit because they feared violence."	"they"	63	"they feared violence"	63	[ "The city councilmen", "The demonstrators" ]	0 (0)	"(Winograd 1972)"
"The city councilmen refused the demonstrators a permit because they advocated violence."	"they"	63	"they advocated violence"	63	[ "The city councilmen", "The demonstrators" ]	1 (1)	"(Winograd 1972)"
"The trophy doesn't fit into the brown suitcase because it is too large."	"it"	55	"it is too large"	55	[ "the trophy", "the suitcase" ]	0 (0)	"Hector Levesque"
"The trophy doesn't fit into the brown suitcase because it is too small."	"it"	55	"it is too small"	55	[ "the trophy", "the suitcase" ]	1 (1)	"Hector Levesque"
"Joan made sure to thank Susan for all the help she had recieved."	"she"	47	"she had received"	47	[ "Joan", "Susan" ]	0 (0)	"Hector Levesque"
"Joan made sure to thank Susan for all the help she had given."	"she"	47	"she had given"	47	[ "Joan", "Susan" ]	1 (1)	"Hector Levesque"
"Paul tried to call George on the phone, but he wasn't successful."	"he"	44	"he wasn't successful."	44	[ "Paul", "George" ]	0 (0)	"Hector Levesque"
"Paul tried to call George on the phone, but he wasn't available."	"he"	44	"he wasn't available."	44	[ "Paul", "George" ]	1 (1)	"Hector Levesque"
"The lawyer asked the witness a question, but he was reluctant to repeat it."	"he"	45	"he was reluctant"	45	[ "the lawyer", "the witness" ]	0 (0)	"Hector Levesque"
"The lawyer asked the witness a question, but he was reluctant to answer it."	"he"	45	"he was reluctant"	45	[ "the lawyer", "the witness" ]	1 (1)	"Hector Levesque"
"The delivery truck zoomed by the school bus because it was going so fast."	"it"	52	"it was going so fast."	52	[ "the delivery truck", "the school bus" ]	0 (0)	"Hector Levesque"
"The delivery truck zoomed by the school bus because it was going so slow."	"it"	52	"it was going so slow."	52	[ "the delivery truck", "the school bus" ]	1 (1)	"Hector Levesque"
"Frank felt vindicated when his longtime rival Bill revealed that he was the winner of the competition."	"he"	65	"he was the winner"	65	[ "Frank", "Bill" ]	0 (0)	"Hector Levesque"
"Frank felt crushed when his longtime rival Bill revealed that he was the winner of the competition."	"he"	62	"he was the winner"	62	[ "Frank", "Bill" ]	1 (1)	"Hector Levesque"
"The man couldn't lift his son because he was so weak."	"he"	38	"he was so weak."	38	[ "The man", "The son" ]	0 (0)	"Hector Levesque"
"The man couldn't lift his son because he was so heavy."	"he"	38	"he was so heavy."	38	[ "The man", "The son" ]	1 (1)	"Hector Levesque"
"The large ball crashed right through the table because it was made of steel."	"it"	55	"it was made of steel."	55	[ "The large ball", "The table" ]	0 (0)	"Hector Levesque"
"The large ball crashed right through the table because it was made of styrofoam."	"it"	55	"it was made of styrofoam."	55	[ "The large ball", "The table" ]	1 (1)	"Hector Levesque"
"John couldn't see the stage with Billy in front of him because he is so short."	"he"	63	"he is so short."	63	[ "John", "Billy" ]	0 (0)	"Hector Levesque"
"John couldn't see the stage with Billy in front of him because he is so tall."	"he"	63	"he is so tall."	63	[ "John", "Billy" ]	1 (1)	"Hector Levesque"
"Tom threw his schoolbag down to Ray after he reached the top of the stairs."	"he"	42	"he reached the top of the stairs."	42	[ "Tom", "Ray" ]	0 (0)	"Hector Levesque"
"Tom threw his schoolbag down to Ray after he reached the bottom of the stairs."	"he"	42	"he reached the bottom of the stairs."	42	[ "Tom", "Ray" ]	1 (1)	"Hector Levesque"
"Although they ran at about the same speed, Sue beat Sally because she had such a good start."	"she"	66	"she had such a good start."	66	[ "Sue", "Sally" ]	0 (0)	"Hector Levesque"
"Although they ran at about the same speed, Sue beat Sally because she had such a bad start."	"she"	66	"she had such a bad start."	66	[ "Sue", "Sally" ]	1 (1)	"Hector Levesque"
"The sculpture rolled off the shelf because it wasn't anchored."	"it"	43	"it wasn't anchored."	43	[ "The sculpture", "The shelf" ]	0 (0)	"Hector Levesque"
"The sculpture rolled off the shelf because it wasn't level."	"it"	43	"it wasn't level."	43	[ "The sculpture", "The shelf" ]	1 (1)	"Hector Levesque"
"Sam's drawing was hung just above Tina's and it did look much better with another one below it."	"it"	45	"it did look much better"	45	[ "Sam's drawing", "Tina's drawing" ]	0 (0)	"Hector Levesque"
"Sam's drawing was hung just above Tina's and it did look much better with another one above it."	"it"	45	"it did look much better"	45	[ "Sam's drawing", "Tina's drawing" ]	1 (1)	"Hector Levesque"
"Anna did a lot better than her good friend Lucy on the test because she had studied so hard."	"she"	68	"she had studied so hard."	68	[ "Anna", "Lucy" ]	0 (0)	"Hector Levesque"
"Anna did a lot worse than her good friend Lucy on the test because she had studied so hard."	"she"	67	"she had studied so hard."	67	[ "Anna", "Lucy" ]	1 (1)	"Hector Levesque"
"The firemen arrived after the police because they were coming from so far away."	"they"	45	"they were coming from so far away."	45	[ "The firemen", "The police" ]	0 (0)	"Hector Levesque"
"The firemen arrived before the police because they were coming from so far away."	"they"	46	"they were coming from so far away."	46	[ "The firemen", "The police" ]	1 (1)	"Hector Levesque"
"Frank was upset with Tom because the toaster he had bought from him didn't work."	"he"	45	"he had bought"	45	[ "Frank", "Tom" ]	0 (0)	"Hector Levesque"
"Frank was upset with Tom because the toaster he had sold him didn't work."	"he"	45	"he had sold"	45	[ "Frank", "Tom" ]	1 (1)	"Hector Levesque"
"Jim yelled at Kevin because he was so upset."	"he"	28	"he was so upset."	28	[ "Jim", "Kevin" ]	0 (0)	"Hector Levesque"
"Jim comforted Kevin because he was so upset."	"he"	28	"he was so upset."	28	[ "Jim", "Kevin" ]	1 (1)	"Hector Levesque"
"The sack of potatoes had been placed above the bag of flour, so it had to be moved first."	"it"	64	"it had to be moved first."	64	[ "The sack of potatoes", "The bag of flour" ]	0 (0)	"Hector Levesque"
"The sack of potatoes had been placed below the bag of flour, so it had to be moved first."	"it"	64	"it had to be moved first."	64	[ "The sack of potatoes", "The bag of flour" ]	1 (1)	"Hector Levesque"
"Pete envies Martin although he is very successful."	"he"	28	"he is very successful."	28	[ "Pete", "Martin" ]	0 (0)	"Ernest Davis"
"Pete envies Martin because he is very successful."	"he"	27	"he is very successful."	27	[ "Pete", "Martin" ]	1 (1)	"Ernest Davis"
"The older students were bullying the younger ones, so we punished them ."	"them"	66	"we punished them"	54	[ "The older students", "The younger students" ]	0 (0)	"Ernest Davis"
"The older students were bullying the younger ones, so we rescued them ."	"them"	65	"we rescued them"	54	[ "The older students", "The younger students" ]	1 (1)	"Ernest Davis"
"I poured water from the bottle into the cup until it was empty."	"it"	50	"it was empty"	50	[ "the bottle", "the cup" ]	0 (0)	"Ernest Davis"
"I poured water from the bottle into the cup until it was full."	"it"	50	"it was full"	50	[ "the bottle", "the cup" ]	1 (1)	"Ernest Davis"
"Susan knows all about Ann's personal problems because she is nosy."	"she"	54	"she is nosy"	54	[ "Susan", "Ann" ]	0 (0)	"Ernest Davis"
"Susan knows all about Ann's personal problems because she is indiscreet."	"she"	54	"she is indiscreet"	54	[ "Susan", "Ann" ]	1 (1)	"Ernest Davis"
"Sid explained his theory to Mark but he couldn't convince him."	"he"	37	"he couldn't convince him."	37	[ "Sid", "Mark" ]	0 (0)	"Ernest Davis"
"Sid explained his theory to Mark but he couldn't understand him."	"he"	37	"he couldn't understand him."	37	[ "Sid", "Mark" ]	1 (1)	"Ernest Davis"
"Susan knew that Ann's son had been in a car accident, so she told her about it."	"she"	57	"she told"	57	[ "Susan", "Ann" ]	0 (0)	"Ernest Davis"
"Susan knew that Ann's son had been in a car accident, because she told her about it."	"she"	62	"she told"	62	[ "Susan", "Ann" ]	1 (1)	"Ernest Davis"
"Joe's uncle can still beat him at tennis, even though he is 30 years younger."	"he"	54	"he is 30 years younger."	54	[ "Joe", "Joe's uncle" ]	0 (0)	"Ernest Davis"
"Joe's uncle can still beat him at tennis, even though he is 30 years older."	"he"	54	"he is 30 years older."	54	[ "Joe", "Joe's uncle" ]	1 (1)	"Ernest Davis"
"The painting in Mark's living room shows an oak tree. It is to the right of the bookcase."	"It"	54	"It is to the right"	54	[ "The painting", "The oak tree" ]	0 (0)	"Ernest Davis"
"The painting in Mark's living room shows an oak tree. It is to the right of a house."	"It"	54	"It is to the right"	54	[ "The painting", "The oak tree" ]	1 (1)	"Ernest Davis"
"There is a gap in the wall. You can see the garden through it ."	"it"	59	"through it"	51	[ "The gap", "The wall" ]	0 (0)	"Ernest Davis"
"There is a gap in the wall. You can see the garden behind it ."	"it"	58	"behind it"	51	[ "The gap", "The wall" ]	1 (1)	"Ernest Davis"
"The drain is clogged with hair. It has to be cleaned."	"It"	32	"It has to be cleaned."	32	[ "The drain", "The hair" ]	0 (0)	"Ernest Davis"
"The drain is clogged with hair. It has to be removed."	"It"	32	"It has to be removed."	32	[ "The drain", "The hair" ]	1 (1)	"Ernest Davis"
"My meeting started at 4:00 and I needed to catch the train at 4:30, so there wasn't much time. Luckily, it was short, so it worked out."	"it"	104	"it was short"	104	[ "The meeting", "The train" ]	0 (0)	"Ernest Davis"
"My meeting started at 4:00 and I needed to catch the train at 4:30, so there wasn't much time. Luckily, it was delayed, so it worked out."	"it"	104	"it was delayed"	104	[ "The meeting", "The train" ]	1 (1)	"Ernest Davis"
"There is a pillar between me and the stage, and I can't see around it ."	"it"	67	"see around it"	56	[ "The pillar", "The stage" ]	0 (0)	"Ernest Davis"
"There is a pillar between me and the stage, and I can't see it ."	"it"	60	"see it"	56	[ "The pillar", "The stage" ]	1 (1)	"Ernest Davis"
"They broadcast an announcement, but a subway came into the station and I couldn't hear it ."	"it"	87	"hear it"	82	[ "The announcement", "The subway" ]	0 (0)	"Ernest Davis"
"They broadcast an announcement, but a subway came into the station and I couldn't hear over it ."	"it"	92	"hear over it"	82	[ "The announcement", "The subway" ]	1 (1)	"Ernest Davis"
"In the middle of the outdoor concert, the rain started falling, but it continued until 10."	"it"	68	"it continued"	68	[ "The concert", "The rain" ]	0 (0)	"Ernest Davis"
"In the middle of the outdoor concert, the rain started falling, and it continued until 10."	"it"	68	"it continued"	68	[ "The concert", "The rain" ]	1 (1)	"Ernest Davis"
"I used an old rag to clean the knife, and then I put it in the trash."	"it"	53	"put it in the trash."	49	[ "The rag", "The knife" ]	0 (0)	"Ernest Davis"
"I used an old rag to clean the knife, and then I put it in the drawer."	"it"	53	"put it in the drawer."	49	[ "The rag", "The knife" ]	1 (1)	"Ernest Davis"
"Ann asked Mary what time the library closes, because she had forgotten."	"she"	53	"she had forgotten."	53	[ "Ann", "Mary" ]	0 (0)	"Ernest Davis"
"Ann asked Mary what time the library closes, but she had forgotten."	"she"	49	"she had forgotten."	49	[ "Ann", "Mary" ]	1 (1)	"Ernest Davis"
"I took the water bottle out of the backpack so that it would be handy."	"it"	52	"it would be handy"	52	[ "The water bottle", "The backpack" ]	0 (0)	"Ernest Davis"
"I took the water bottle out of the backpack so that it would be lighter."	"it"	52	"it would be lighter"	52	[ "The water bottle", "The backpack" ]	1 (1)	"Ernest Davis"
"I couldn't put the pot on the shelf because it was too tall."	"it"	44	"it was too tall."	44	[ "The pot", "The shelf" ]	0 (0)	"Ernest Davis"
"I couldn't put the pot on the shelf because it was too high."	"it"	44	"it was too high."	44	[ "The pot", "The shelf" ]	1 (1)	"Ernest Davis"
"I'm sure that my map will show this building; it is very good."	"it"	46	"it is very good"	46	[ "The map", "The building" ]	0 (0)	"Ernest Davis"
"I'm sure that my map will show this building; it is very famous."	"it"	46	"it is very famous"	46	[ "The map", "The building" ]	1 (1)	"Ernest Davis"
"Bob paid for Charlie's college education. He is very generous."	"He"	42	"He is very generous."	42	[ "Bob", "Charlie" ]	0 (0)	"Ernest Davis"
"Bob paid for Charlie's college education. He is very grateful."	"He"	42	"He is very grateful."	42	[ "Bob", "Charlie" ]	1 (1)	"Ernest Davis"
"Bob paid for Charlie's college education, but now Charlie acts as though it never happened. He is very hurt."	"He"	92	"He is very hurt."	92	[ "Bob", "Charlie" ]	0 (0)	"Ernest Davis"
"Bob paid for Charlie's college education, but now Charlie acts as though it never happened. He is very ungrateful."	"He"	92	"He is very ungrateful."	92	[ "Bob", "Charlie" ]	1 (1)	"Ernest Davis"
"Bob was playing cards with Adam and was way ahead. If Adam hadn't had a sudden run of good luck, he would have won."	"he"	97	"he would have won."	97	[ "Bob", "Adam" ]	0 (0)	"Ernest Davis"
"Bob was playing cards with Adam and was way ahead. If Adam hadn't had a sudden run of good luck, he would have lost."	"he"	97	"he would have lost."	97	[ "Bob", "Adam" ]	1 (1)	"Ernest Davis"
"Adam can't leave work here until Bob arrives to replace him. If Bob had left home for work on time, he would be gone by this time."	"he"	100	"he would be gone"	100	[ "Adam", "Bob" ]	0 (0)	"Ernest Davis"
"Adam can't leave work here until Bob arrives to replace him. If Bob had left home for work on time, he would be here by this time."	"he"	100	"he would be here"	100	[ "Adam", "Bob" ]	1 (1)	"Ernest Davis"
"If the con artist has succeeded in fooling Sam, he would have gotten a lot of money."	"he"	48	"he would have gotten a lot of money."	48	[ "The con artist", "Sam" ]	0 (0)	"Ernest Davis"
"If the con artist has succeeded in fooling Sam, he would have lost a lot of money."	"he"	48	"he would have lost a lot of money."	48	[ "The con artist", "Sam" ]	1 (1)	"Ernest Davis"
"It was a summer afternoon, and the dog was sitting in the middle of the lawn. After a while, it got up and moved to a spot under the tree, because it was hot."	"it"	147	"it was hot"	147	[ "The dog", "The spot under the tree" ]	0 (0)	"Ernest Davis"
"It was a summer afternoon, and the dog was sitting in the middle of the lawn. After a while, it got up and moved to a spot under the tree, because it was cooler."	"it"	147	"it was cooler"	147	[ "The dog", "The spot under the tree" ]	1 (1)	"Ernest Davis"
"The cat was lying by the mouse hole waiting for the mouse, but it was too impatient."	"it"	63	"it was too impatient."	63	[ "The cat", "The mouse" ]	0 (0)	"Ernest Davis"
"The cat was lying by the mouse hole waiting for the mouse, but it was too cautious."	"it"	63	"it was too cautious."	63	[ "The cat", "The mouse" ]	1 (1)	"Ernest Davis"
"Anne gave birth to a daughter last month. She is a very charming woman."	"She"	42	"She is a very charming woman."	42	[ "Anne", "Anne's daughter" ]	0 (0)	"Ernest Davis"
"Anne gave birth to a daughter last month. She is a very charming baby."	"She"	42	"She is a very charming baby."	42	[ "Anne", "Anne's daughter" ]	1 (1)	"Ernest Davis"
"Alice tried frantically to stop her daughter from chatting at the party, leaving us to wonder why she was behaving so strangely."	"she"	98	"she was behaving so strangely."	98	[ "Alice", "Alice's daughter" ]	0 (0)	"Ernest Davis"
"Alice tried frantically to stop her daughter from barking at the party, leaving us to wonder why she was behaving so strangely."	"she"	97	"she was behaving so strangely."	97	[ "Alice", "Alice's daughter" ]	1 (1)	"Ernest Davis"
"I saw Jim yelling at some guy in a military uniform with a huge red beard. I don't know why he was, but he looked very unhappy."	"he"	92	"why he was"	88	[ "Jim", "the guy in uniform" ]	0 (0)	"Ernest Davis"
"I saw Jim yelling at some guy in a military uniform with a huge red beard. I don't know who he was, but he looked very unhappy."	"he"	92	"who he was"	88	[ "Jim", "the guy in uniform" ]	1 (1)	"Ernest Davis"
"The fish ate the worm. It was hungry."	"It"	23	"It was hungry."	23	[ "The fish", "The worm" ]	0 (0)	"Ernest Davis"
"The fish ate the worm. It was tasty."	"It"	23	"It was tasty."	23	[ "The fish", "The worm" ]	1 (1)	"Ernest Davis"
"I was trying to open the lock with the key, but someone had filled the keyhole with chewing gum, and I couldn't get it in."	"it"	116	"I couldn't get it in."	101	[ "The key", "The chewing gum" ]	0 (0)	"Ernest Davis"
"I was trying to open the lock with the key, but someone had filled the keyhole with chewing gum, and I couldn't get it out."	"it"	116	"I couldn't get it out."	101	[ "The key", "The chewing gum" ]	1 (1)	"Ernest Davis"

Dataset Card for The Winograd Schema Challenge

Dataset Summary

A Winograd schema is a pair of sentences that differ in only one or two words and that contain an ambiguity that is resolved in opposite ways in the two sentences and requires the use of world knowledge and reasoning for its resolution. The schema takes its name from a well-known example by Terry Winograd:

The city councilmen refused the demonstrators a permit because they [feared/advocated] violence.

If the word is feared'', then they'' presumably refers to the city council; if it is advocated'' then they'' presumably refers to the demonstrators.

Supported Tasks and Leaderboards

From the official webpage:

A contest, entitled the Winograd Schema Challenge was run once, in 2016. At that time, there was a cash prize offered for achieving human-level performance in the contest. Since then, the sponsor has withdrawn; therefore NO CASH PRIZES CAN BE OFFERED OR WILL BE AWARDED FOR ANY KIND OF PERFORMANCE OR ACHIEVEMENT ON THIS CHALLENGE.

Languages

The dataset is in English.

Translation of 12 WSs into Chinese (translated by Wei Xu).

Translations into Japanese, by Soichiro Tanaka, Rafal Rzepka, and Shiho Katajima
**Translation changing English names to Japanese **PDF HTML
Translation preserving English names PDF HTML

Translation into French, by Pascal Amsili and Olga Seminck

Winograd Schemas in Portuguese by Gabriela Melo, Vinicius Imaizumi, and Fábio Cozman.

Mandarinograd: A Chinese Collection of Winograd Schemas by Timothée Bernard and Ting Han, LREC-2020.

Dataset Structure

Data Instances

Each instance contains a text passage with a designated pronoun and two possible answers indicating which entity in the passage the pronoun represents. An example instance looks like the following:

{
  'label': 0,
  'options': ['The city councilmen', 'The demonstrators'],
  'pronoun': 'they',
  'pronoun_loc': 63,
  'quote': 'they feared violence',
  'quote_loc': 63,
  'source': '(Winograd 1972)',
  'text': 'The city councilmen refused the demonstrators a permit because they feared violence.'
}

Data Fields

text (str): The text sequence
options (list[str]): The two entity options that the pronoun may be referring to
label (int): The index of the correct option in the options field
pronoun (str): The pronoun in the sequence to be resolved
pronoun_loc (int): The starting position of the pronoun in the sequence
quote (str): The substr with the key action or context surrounding the pronoun
quote_loc (int): The starting position of the quote in the sequence
source (str): A description of the source who contributed the example

Data Splits

Only a test split is included.

Dataset Creation

Curation Rationale

The Winograd Schema Challenge was proposed as an automated evaluation of an AI system's commonsense linguistic understanding. From the webpage:

The strengths of the challenge are that it is clear-cut, in that the answer to each schema is a binary choice; vivid, in that it is obvious to non-experts that a program that fails to get the right answers clearly has serious gaps in its understanding; and difficult, in that it is far beyond the current state of the art.

Source Data

Initial Data Collection and Normalization

This data was manually written by experts such that the schemas are:

easily disambiguated by the human reader (ideally, so easily that the reader does not even notice that there is an ambiguity);
not solvable by simple techniques such as selectional restrictions;
Google-proof; that is, there is no obvious statistical test over text corpora that will reliably disambiguate these correctly.

Who are the source language producers?

This dataset has grown over time, and so was produced by a variety of lingustic and AI researchers. See the source field for the source of each instance.

Annotations

Annotation process

Annotations are produced by the experts who construct the examples.

Who are the annotators?

See above.

Personal and Sensitive Information

[More Information Needed]

Considerations for Using the Data

Social Impact of Dataset

[More Information Needed]

Discussion of Biases

[More Information Needed]

Other Known Limitations

[More Information Needed]

Additional Information

Dataset Curators

This dataset has grown over time, and so was produced by a variety of lingustic and AI researchers. See the source field for the source of each instance.

Licensing Information

This work is licensed under a Creative Commons Attribution 4.0 International License.

Citation Information

The Winograd Schema Challenge including many of the examples here was proposed by Levesque et al 2012:

@inproceedings{levesque2012winograd,
  title={The winograd schema challenge},
  author={Levesque, Hector and Davis, Ernest and Morgenstern, Leora},
  booktitle={Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning},
  year={2012},
  organization={Citeseer}
}