diff --git a/transformers/fine-tune-QA.ipynb b/transformers/fine-tune-QA.ipynb index 5f26e10..474ae3e 100644 --- a/transformers/fine-tune-QA.ipynb +++ b/transformers/fine-tune-QA.ipynb @@ -17,7 +17,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 2, "metadata": { "id": "zVvslsfMIrIh" }, @@ -48,7 +48,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 3, "metadata": { "id": "IreSlFmlIrIm" }, @@ -59,7 +59,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -126,78 +126,7 @@ "id": "s_AY1ATSIrIq", "outputId": "fd0578d1-8895-443d-b56f-5908de9f1b6b" }, - "outputs": [ - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "ad7fc91eff1e463b8585cfa6406b5c64", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "Downloading readme: 0%| | 0.00/7.83k [00:00\n", " \n", " 0\n", - " 572673df5951b619008f730b\n", - " Professional_wrestling\n", - " Though they have not had the level of exposure as other wrestlers, bears have long been a part of professional wrestling. Usually declawed and muzzled, they often wrestled shoot matches against audience members, offered a cash reward if they could pin the bear. They also wrestled professionals in worked, often battle royal or handicap, matches (usually booked so the bear won). Though they have wrestled around the world and continue to do so, wrestling bears enjoyed their greatest popularity in the Southern United States, during the 1960s and 1970s. The practice of bear wrestling has met strong opposition from animal rights activists in recent decades, contributing to its lack of mainstream acceptance. As of 2006, it is banned in 20 U.S. states. Perhaps the most famous wrestling bears are Ginger, Victor, Hercules and Terrible Ted.\n", - " How many states have banned bear wrestling as of 2006?\n", - " {'text': ['20'], 'answer_start': [739]}\n", + " 572794e2dd62a815002ea11a\n", + " Crucifixion_of_Jesus\n", + " Sextus Julius Africanus further refers to the writings of historian Thallus: \"This darkness Thallus, in the third book of his History, calls, as appears to me without reason, an eclipse of the sun. For the Hebrews celebrate the passover on the 14th day according to the moon, and the passion of our Saviour falls on the day before the passover; but an eclipse of the sun takes place only when the moon comes under the sun.\" Christian apologist Tertullian believed the event was documented in the Roman archives.\n", + " Which historian was referenced for the reports?\n", + " {'text': ['Thallus'], 'answer_start': [68]}\n", " \n", " \n", " 1\n", - " 5728c8e3ff5b5019007da6a0\n", - " London\n", - " The pilgrims in Geoffrey Chaucer's late 14th-century Canterbury Tales set out for Canterbury from London – specifically, from the Tabard inn, Southwark. William Shakespeare spent a large part of his life living and working in London; his contemporary Ben Jonson was also based there, and some of his work—most notably his play The Alchemist—was set in the city. A Journal of the Plague Year (1722) by Daniel Defoe is a fictionalisation of the events of the 1665 Great Plague. Later important depictions of London from the 19th and early 20th centuries are Dickens' novels, and Arthur Conan Doyle's Sherlock Holmes stories. Modern writers pervasively influenced by the city include Peter Ackroyd, author of a \"biography\" of London, and Iain Sinclair, who writes in the genre of psychogeography.\n", - " What famous Elizabethan playwright spent much of his life in London?\n", - " {'text': ['William Shakespeare'], 'answer_start': [153]}\n", + " 56d4d5ef2ccc5a1400d8328a\n", + " Beyoncé\n", + " In 2015 Beyoncé signed an open letter which the ONE Campaign had been collecting signatures for; the letter was addressed to Angela Merkel and Nkosazana Dlamini-Zuma, urging them to focus on women as they serve as the head of the G7 in Germany and the AU in South Africa respectively, which will start to set the priorities in development funding before a main UN summit in September 2015 that will establish new development goals for the generation.\n", + " What did the letter want the two recipients to focus on?\n", + " {'text': ['women'], 'answer_start': [191]}\n", " \n", " \n", " 2\n", - " 571a69014faf5e1900b8a992\n", - " Ashkenazi_Jews\n", - " A 2010 study by Bray et al., using SNP microarray techniques and linkage analysis found that when assuming Druze and Palestinian Arab populations to represent the reference to world Jewry ancestor genome, between 35 to 55 percent of the modern Ashkenazi genome can possibly be of European origin, and that European \"admixture is considerably higher than previous estimates by studies that used the Y chromosome\" with this reference point. Assuming this reference point the linkage disequilibrium in the Ashkenazi Jewish population was interpreted as \"matches signs of interbreeding or 'admixture' between Middle Eastern and European populations\". On the Bray et al. tree, Ashkenazi Jews were found to be a genetically more divergent population than Russians, Orcadians, French, Basques, Italians, Sardinians and Tuscans. The study also observed that Ashkenazim are more diverse than their Middle Eastern relatives, which was counterintuitive because Ashkenazim are supposed to be a subset, not a superset, of their assumed geographical source population. Bray et al. therefore postulate that these results reflect not the population antiquity but a history of mixing between genetically distinct populations in Europe. However, it's possible that the relaxation of marriage prescription in the ancestors of Ashkenazim that drove their heterozygosity up, while the maintenance of the FBD rule in native Middle Easterners have been keeping their heterozygosity values in check. Ashkenazim distinctiveness as found in the Bray et al. study, therefore, may come from their ethnic endogamy (ethnic inbreeding), which allowed them to \"mine\" their ancestral gene pool in the context of relative reproductive isolation from European neighbors, and not from clan endogamy (clan inbreeding). Consequently, their higher diversity compared to Middle Easterners stems from the latter's marriage practices, not necessarily from the former's admixture with Europeans.\n", - " Were the Ashkenazim thought to be a subset or superset of their assumed geographical source population?\n", - " {'text': ['Ashkenazim are supposed to be a subset, not a superset'], 'answer_start': [950]}\n", + " 57309c8e2461fd1900a9cf09\n", + " Greeks\n", + " As of 2007, Greece had the eighth highest percentage of tertiary enrollment in the world (with the percentages for female students being higher than for male) while Greeks of the Diaspora are equally active in the field of education. Hundreds of thousands of Greek students attend western universities every year while the faculty lists of leading Western universities contain a striking number of Greek names. Notable modern Greek scientists of modern times include Dimitrios Galanos, Georgios Papanikolaou (inventor of the Pap test), Nicholas Negroponte, Constantin Carathéodory, Manolis Andronikos, Michael Dertouzos, John Argyris, Panagiotis Kondylis, John Iliopoulos (2007 Dirac Prize for his contributions on the physics of the charm quark, a major contribution to the birth of the Standard Model, the modern theory of Elementary Particles), Joseph Sifakis (2007 Turing Award, the \"Nobel Prize\" of Computer Science), Christos Papadimitriou (2002 Knuth Prize, 2012 Gödel Prize), Mihalis Yannakakis (2005 Knuth Prize) and Dimitri Nanopoulos.\n", + " According to recent research who is enrolled in the Universities in Greece in higher number men or women ?\n", + " {'text': ['the percentages for female students being higher than for male'], 'answer_start': [95]}\n", " \n", " \n", " 3\n", - " 572f9c93b2c2fd140056824c\n", - " Hyderabad\n", - " Hyderabad emerged as the foremost centre of culture in India with the decline of the Mughal Empire. After the fall of Delhi in 1857, the migration of performing artists to the city particularly from the north and west of the Indian sub continent, under the patronage of the Nizam, enriched the cultural milieu. This migration resulted in a mingling of North and South Indian languages, cultures and religions, which has since led to a co-existence of Hindu and Muslim traditions, for which the city has become noted.:viii A further consequence of this north–south mix is that both Telugu and Urdu are official languages of Telangana. The mixing of religions has also resulted in many festivals being celebrated in Hyderabad such as Ganesh Chaturthi, Diwali and Bonalu of Hindu tradition and Eid ul-Fitr and Eid al-Adha by Muslims.\n", - " What major event in India in 1857 caused migration of peformance artists into Hyderabad?\n", - " {'text': ['the fall of Delhi'], 'answer_start': [106]}\n", + " 56cf5e35aab44d1400b89148\n", + " Frédéric_Chopin\n", + " In October 1810, six months after Fryderyk's birth, the family moved to Warsaw, where his father acquired a post teaching French at the Warsaw Lyceum, then housed in the Saxon Palace. Fryderyk lived with his family in the Palace grounds. The father played the flute and violin; the mother played the piano and gave lessons to boys in the boarding house that the Chopins kept. Chopin was of slight build, and even in early childhood was prone to illnesses.\n", + " Where did Chopin's father get a teaching position?\n", + " {'text': ['Warsaw Lyceum'], 'answer_start': [136]}\n", " \n", " \n", " 4\n", - " 572670f2f1498d1400e8dfb4\n", - " Florida\n", - " The first post-Reconstruction era Republican elected to Congress from Florida was William C. Cramer in 1954 from Pinellas County on the Gulf Coast, where demographic changes were underway. In this period, African Americans were still disenfranchised by the state's constitution and discriminatory practices; in the 19th century they had made up most of the Republican Party. Cramer built a different Republican Party in Florida, attracting local white conservatives and transplants from northern and midwestern states. In 1966 Claude R. Kirk, Jr. was elected as the first post-Reconstruction Republican governor, in an upset election. In 1968 Edward J. Gurney, also a white conservative, was elected as the state's first post-reconstruction Republican US Senator. In 1970 Democrats took the governorship and the open US Senate seat, and maintained dominance for years.\n", - " The first post- reconstruction era republican elected to florida\n", - " {'text': ['William C. Cramer in 1954 from Pinellas County'], 'answer_start': [82]}\n", + " 572acab534ae481900deaceb\n", + " John_Kerry\n", + " In addition to the sports he played at Yale, Kerry is described by Sports Illustrated, among others, as an \"avid cyclist\", primarily riding on a road bike. Prior to his presidential bid, Kerry was known to have participated in several long-distance rides (centuries). Even during his many campaigns, he was reported to have visited bicycle stores in both his home state and elsewhere. His staff requested recumbent stationary bikes for his hotel rooms. He has also been a snowboarder, windsurfer, and sailor.\n", + " Who said Kerry is an 'avid cyclist'?\n", + " {'text': ['Sports Illustrated'], 'answer_start': [67]}\n", " \n", " \n", " 5\n", - " 56fc3e072603e7140040a03d\n", - " Phonology\n", - " In a course at the LSA summer institute in 1991, Alan Prince and Paul Smolensky developed optimality theory—an overall architecture for phonology according to which languages choose a pronunciation of a word that best satisfies a list of constraints ordered by importance; a lower-ranked constraint can be violated when the violation is necessary in order to obey a higher-ranked constraint. The approach was soon extended to morphology by John McCarthy and Alan Prince, and has become a dominant trend in phonology. The appeal to phonetic grounding of constraints and representational elements (e.g. features) in various approaches has been criticized by proponents of 'substance-free phonology', especially Mark Hale and Charles Reiss.\n", - " When did Prince and Smolensky develop their theory?\n", - " {'text': ['1991'], 'answer_start': [43]}\n", + " 572f554ea23a5019007fc567\n", + " Central_Intelligence_Agency\n", + " The closest links of the U.S. IC to other foreign intelligence agencies are to Anglophone countries: Australia, Canada, New Zealand, and the United Kingdom. There is a special communications marking that signals that intelligence-related messages can be shared with these four countries. An indication of the United States' close operational cooperation is the creation of a new message distribution label within the main U.S. military communications network. Previously, the marking of NOFORN (i.e., No Foreign Nationals) required the originator to specify which, if any, non-U.S. countries could receive the information. A new handling caveat, USA/AUS/CAN/GBR/NZL Five Eyes, used primarily on intelligence messages, gives an easier way to indicate that the material can be shared with Australia, Canada, United Kingdom, and New Zealand.\n", + " What is the special marking that shows messages can be shared with these other five countries?\n", + " {'text': ['USA/AUS/CAN/GBR/NZL Five Eyes'], 'answer_start': [646]}\n", " \n", " \n", " 6\n", - " 5726bc0b708984140094cf8d\n", - " Mali\n", - " Anti-government protests in 1991 led to a coup, a transitional government, and a new constitution. Opposition to the corrupt and dictatorial regime of General Moussa Traoré grew during the 1980s. During this time strict programs, imposed to satisfy demands of the International Monetary Fund, brought increased hardship upon the country's population, while elites close to the government supposedly lived in growing wealth. Peaceful student protests in January 1991 were brutally suppressed, with mass arrests and torture of leaders and participants. Scattered acts of rioting and vandalism of public buildings followed, but most actions by the dissidents remained nonviolent.\n", - " In what year did Mali get a new constitution?\n", - " {'text': ['1991'], 'answer_start': [28]}\n", + " 56f9eaae8f12f3190062fff3\n", + " High-definition_television\n", + " In addition, recording and reproducing an HDTV signal was a significant technical challenge in the early years of HDTV (Sony HDVS). Japan remained the only country with successful public broadcasting of analog HDTV, with seven broadcasters sharing a single channel.\n", + " What was particularly challenging in the early years of HDTV?\n", + " {'text': ['recording and reproducing an HDTV signal'], 'answer_start': [13]}\n", " \n", " \n", " 7\n", - " 57268b01708984140094c98e\n", - " Greece\n", - " The Greek national football team, ranking 12th in the world in 2014 (and having reached a high of 8th in the world in 2008 and 2011), were crowned European Champions in Euro 2004 in one of the biggest upsets in the history of the sport and became one of the most successful national teams in European football, being one of only nine national teams to have won the UEFA European Championship. The Greek Super League is the highest professional football league in the country comprising eighteen teams. The most successful are Olympiacos, Panathinaikos, AEK Athens and PAOK.\n", - " Greek was crowned the European champions in what event?\n", - " {'text': ['Euro 2004'], 'answer_start': [169]}\n", + " 56e0f229231d4119001ac4b5\n", + " Space_Race\n", + " Three weeks later, on May 5, 1961, Alan Shepard became the first American in space, launched in a ballistic trajectory on Mercury-Redstone 3, in a spacecraft he named Freedom 7. Though he did not achieve orbit like Gagarin, he was the first person to exercise manual control over his spacecraft's attitude and retro-rocket firing. After his successful return, Shepard was celebrated as a national hero, honored with parades in Washington, New York and Los Angeles, and received the NASA Distinguished Service Medal from President John F. Kennedy.\n", + " The first person to have control over driving their spacecraft was whom?\n", + " {'text': ['Alan Shepard'], 'answer_start': [35]}\n", " \n", " \n", " 8\n", - " 57269adedd62a815002e8ad6\n", - " Korean_War\n", - " Acting on State Secretary Acheson's recommendation, President Truman ordered General MacArthur to transfer matériel to the Army of the Republic of Korea while giving air cover to the evacuation of U.S. nationals. The President disagreed with advisers who recommended unilateral U.S. bombing of the North Korean forces, and ordered the US Seventh Fleet to protect the Republic of China (Taiwan), whose government asked to fight in Korea. The United States denied ROC's request for combat, lest it provoke a communist Chinese retaliation. Because the United States had sent the Seventh Fleet to \"neutralize\" the Taiwan Strait, Chinese premier Zhou Enlai criticized both the UN and U.S. initiatives as \"armed aggression on Chinese territory.\"\n", - " What general provided material to the Republic of Korea and evacuated US nationals?\n", - " {'text': ['General MacArthur'], 'answer_start': [77]}\n", + " 56df705a5ca0a614008f9a36\n", + " Christian\n", + " Korean still uses 기독교도, Kidok-kyo-do for \"Christian\", though the Greek form Kurisudo 그리스도 has now replaced the old Sino-Korean Kidok, which refers to Christ himself.\n", + " Which term referred to Jesus himself?\n", + " {'text': ['Sino-Korean Kidok'], 'answer_start': [115]}\n", " \n", " \n", " 9\n", - " 56cf4815aab44d1400b88f1c\n", - " Spectre_(2015_film)\n", - " Filming started in Austria in December 2014, with production taking in the area around Sölden—including the Ötztal Glacier Road, Rettenbach glacier and the adjacent ski resort and cable car station—and Obertilliach and Lake Altaussee, before concluding in February 2015. Scenes filmed in Austria centred on the Ice Q Restaurant, standing in for the fictional Hoffler Klinik, a private medical clinic in the Austrian Alps. Filming included an action scene featuring a Land Rover Defender Bigfoot and a Range Rover Sport. Production was temporarily halted first by an injury to Craig, who sprained his knee whilst shooting a fight scene, and later by an accident involving a filming vehicle that saw three crew members injured, at least one of them seriously.\n", - " Where was the Hoffler Klinik supposed to be located?\n", - " {'text': ['Austrian Alps.'], 'answer_start': [407]}\n", + " 572fcc32a23a5019007fc9fc\n", + " Armenia\n", + " The economy relies heavily on investment and support from Armenians abroad. Before independence, Armenia's economy was largely industry-based – chemicals, electronics, machinery, processed food, synthetic rubber, and textile – and highly dependent on outside resources. The republic had developed a modern industrial sector, supplying machine tools, textiles, and other manufactured goods to sister republics in exchange for raw materials and energy. Recently, the Intel Corporation agreed to open a research center in Armenia, in addition to other technology companies, signalling the growth of the technology industry in Armenia.\n", + " As opposed to an industry-based economy, what industry is Armenia moving to?\n", + " {'text': ['technology industry'], 'answer_start': [600]}\n", " \n", " \n", "" @@ -464,7 +393,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 7, "metadata": { "id": "eXNLu_-nIrJI" }, @@ -486,7 +415,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 8, "metadata": {}, "outputs": [], "source": [ @@ -505,7 +434,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 9, "metadata": { "id": "a5hBlsrHIrJL", "outputId": "acdaa98a-a8cd-4a20-89b8-cc26437bbe90" @@ -517,7 +446,7 @@ "{'input_ids': [101, 2054, 2003, 2115, 2171, 1029, 102, 2026, 2171, 2003, 25353, 22144, 2378, 1012, 102], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}" ] }, - "execution_count": 10, + "execution_count": 9, "metadata": {}, "output_type": "execute_result" } @@ -541,7 +470,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 14, "metadata": {}, "outputs": [], "source": [ @@ -562,7 +491,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 21, "metadata": {}, "outputs": [], "source": [ @@ -575,7 +504,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 22, "metadata": {}, "outputs": [ { @@ -584,7 +513,7 @@ "396" ] }, - "execution_count": 14, + "execution_count": 22, "metadata": {}, "output_type": "execute_result" } @@ -602,7 +531,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 23, "metadata": {}, "outputs": [ { @@ -611,7 +540,7 @@ "384" ] }, - "execution_count": 16, + "execution_count": 23, "metadata": {}, "output_type": "execute_result" } @@ -635,7 +564,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 24, "metadata": {}, "outputs": [], "source": [ @@ -658,7 +587,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 25, "metadata": {}, "outputs": [ { @@ -667,7 +596,7 @@ "[384, 157]" ] }, - "execution_count": 18, + "execution_count": 25, "metadata": {}, "output_type": "execute_result" } @@ -685,7 +614,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 26, "metadata": {}, "outputs": [ { @@ -715,7 +644,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 27, "metadata": {}, "outputs": [ { @@ -750,7 +679,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 28, "metadata": {}, "outputs": [ { @@ -769,7 +698,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 29, "metadata": {}, "outputs": [ { @@ -788,7 +717,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 30, "metadata": {}, "outputs": [ { @@ -797,7 +726,7 @@ "\"How many wins does the Notre Dame men's basketball team have?\"" ] }, - "execution_count": 25, + "execution_count": 30, "metadata": {}, "output_type": "execute_result" } @@ -820,7 +749,7 @@ }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 31, "metadata": {}, "outputs": [ { @@ -838,7 +767,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 32, "metadata": {}, "outputs": [ { @@ -889,7 +818,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 33, "metadata": {}, "outputs": [ { @@ -920,7 +849,7 @@ }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 10, "metadata": {}, "outputs": [], "source": [ @@ -942,7 +871,7 @@ }, { "cell_type": "code", - "execution_count": 31, + "execution_count": 12, "metadata": {}, "outputs": [], "source": [ @@ -1039,7 +968,7 @@ }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 15, "metadata": { "id": "DDtsaJeVIrJT", "outputId": "aa4734bf-4ef5-4437-9948-2c16363da719" @@ -1048,7 +977,7 @@ { "data": { "application/vnd.jupyter.widget-view+json": { - "model_id": "fa7014558b1240b3a16d6ea754784757", + "model_id": "0901378728e3407f875c16fa9a0b8d92", "version_major": 2, "version_minor": 0 }, @@ -1062,7 +991,7 @@ { "data": { "application/vnd.jupyter.widget-view+json": { - "model_id": "11ba79be8af24e7fa52073b44a58b275", + "model_id": "2f6148a5774e4c3ab720ed0a3803ce30", "version_major": 2, "version_minor": 0 }, @@ -1104,7 +1033,7 @@ }, { "cell_type": "code", - "execution_count": 33, + "execution_count": 30, "metadata": { "id": "TlqNaB8jIrJW", "outputId": "84916cf3-6e6c-47f3-d081-032ec30a4132" @@ -1114,7 +1043,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "Some weights of DistilBertForQuestionAnswering were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['qa_outputs.weight', 'qa_outputs.bias']\n", + "Some weights of DistilBertForQuestionAnswering were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['qa_outputs.bias', 'qa_outputs.weight']\n", "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n" ] } @@ -1200,7 +1129,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "Detected kernel version 4.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n" + "Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n" ] } ], @@ -1270,7 +1199,7 @@ "
\n", " \n", " \n", - " [4152/4152 2:23:19, Epoch 3/3]\n", + " [4152/4152 2:23:13, Epoch 3/3]\n", "
\n", " \n", " \n", @@ -1283,18 +1212,18 @@ " \n", " \n", " \n", - " \n", - " \n", + " \n", + " \n", " \n", " \n", " \n", - " \n", - " \n", + " \n", + " \n", " \n", " \n", " \n", - " \n", - " \n", + " \n", + " \n", " \n", " \n", "
11.4911001.2494411.5112001.266460
21.1088001.1616711.1331001.192939
30.9757001.1587660.9969001.177918

" @@ -1306,24 +1235,10 @@ "metadata": {}, "output_type": "display_data" }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "Checkpoint destination directory models/distilbert-base-uncased-finetuned-squad/checkpoint-500 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n", - "Checkpoint destination directory models/distilbert-base-uncased-finetuned-squad/checkpoint-1000 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n", - "Checkpoint destination directory models/distilbert-base-uncased-finetuned-squad/checkpoint-1500 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n", - "Checkpoint destination directory models/distilbert-base-uncased-finetuned-squad/checkpoint-2000 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n", - "Checkpoint destination directory models/distilbert-base-uncased-finetuned-squad/checkpoint-2500 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n", - "Checkpoint destination directory models/distilbert-base-uncased-finetuned-squad/checkpoint-3000 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n", - "Checkpoint destination directory models/distilbert-base-uncased-finetuned-squad/checkpoint-3500 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n", - "Checkpoint destination directory models/distilbert-base-uncased-finetuned-squad/checkpoint-4000 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n" - ] - }, { "data": { "text/plain": [ - "TrainOutput(global_step=4152, training_loss=1.3038662743246854, metrics={'train_runtime': 8602.4737, 'train_samples_per_second': 30.872, 'train_steps_per_second': 0.483, 'total_flos': 2.602335381127373e+16, 'train_loss': 1.3038662743246854, 'epoch': 3.0})" + "TrainOutput(global_step=4152, training_loss=1.325681671701185, metrics={'train_runtime': 8596.3773, 'train_samples_per_second': 30.893, 'train_steps_per_second': 0.483, 'total_flos': 2.602335381127373e+16, 'train_loss': 1.325681671701185, 'epoch': 3.0})" ] }, "execution_count": 37, @@ -1369,18 +1284,19 @@ }, { "cell_type": "code", - "execution_count": 39, + "execution_count": 2, "metadata": {}, "outputs": [ { - "data": { - "text/plain": [ - "odict_keys(['loss', 'start_logits', 'end_logits'])" - ] - }, - "execution_count": 39, - "metadata": {}, - "output_type": "execute_result" + "ename": "NameError", + "evalue": "name 'trainer' is not defined", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", + "Cell \u001b[0;32mIn[2], line 3\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mtorch\u001b[39;00m\n\u001b[0;32m----> 3\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m batch \u001b[38;5;129;01min\u001b[39;00m \u001b[43mtrainer\u001b[49m\u001b[38;5;241m.\u001b[39mget_eval_dataloader():\n\u001b[1;32m 4\u001b[0m \u001b[38;5;28;01mbreak\u001b[39;00m\n\u001b[1;32m 5\u001b[0m batch \u001b[38;5;241m=\u001b[39m {k: v\u001b[38;5;241m.\u001b[39mto(trainer\u001b[38;5;241m.\u001b[39margs\u001b[38;5;241m.\u001b[39mdevice) \u001b[38;5;28;01mfor\u001b[39;00m k, v \u001b[38;5;129;01min\u001b[39;00m batch\u001b[38;5;241m.\u001b[39mitems()}\n", + "\u001b[0;31mNameError\u001b[0m: name 'trainer' is not defined" + ] } ], "source": [ @@ -1429,15 +1345,15 @@ { "data": { "text/plain": [ - "(tensor([ 46, 57, 78, 43, 118, 108, 72, 35, 108, 34, 73, 41, 80, 91,\n", + "(tensor([ 46, 57, 78, 43, 118, 15, 72, 35, 15, 34, 73, 41, 80, 91,\n", " 156, 35, 83, 91, 80, 58, 77, 31, 42, 53, 41, 35, 42, 77,\n", - " 11, 44, 27, 133, 66, 40, 87, 44, 43, 41, 127, 26, 28, 33,\n", + " 11, 44, 27, 133, 66, 40, 87, 44, 43, 83, 127, 26, 28, 33,\n", " 87, 127, 95, 25, 43, 132, 42, 29, 44, 46, 24, 44, 65, 58,\n", " 81, 14, 59, 72, 25, 36, 57, 43], device='cuda:0'),\n", " tensor([ 47, 58, 81, 44, 118, 109, 75, 37, 109, 36, 76, 42, 83, 94,\n", " 158, 35, 83, 94, 83, 60, 80, 31, 43, 54, 42, 35, 43, 80,\n", - " 13, 45, 28, 133, 66, 41, 89, 45, 44, 42, 127, 27, 30, 34,\n", - " 32, 127, 97, 26, 44, 132, 43, 30, 45, 47, 25, 45, 65, 59,\n", + " 13, 45, 28, 133, 66, 41, 89, 45, 44, 85, 127, 27, 30, 34,\n", + " 89, 127, 97, 26, 44, 132, 43, 30, 45, 47, 25, 45, 65, 59,\n", " 81, 14, 60, 72, 25, 36, 58, 43], device='cuda:0'))" ] }, @@ -1588,7 +1504,7 @@ { "data": { "application/vnd.jupyter.widget-view+json": { - "model_id": "9dbc85bdcee9457891a7cc94d54fcd8e", + "model_id": "3123dc78ff554cf99771673307cd62e7", "version_major": 2, "version_minor": 0 }, @@ -1617,7 +1533,7 @@ }, { "cell_type": "code", - "execution_count": 46, + "execution_count": 48, "metadata": {}, "outputs": [ { @@ -1644,7 +1560,7 @@ }, { "cell_type": "code", - "execution_count": 47, + "execution_count": 50, "metadata": {}, "outputs": [], "source": [ @@ -1669,7 +1585,7 @@ }, { "cell_type": "code", - "execution_count": 48, + "execution_count": 51, "metadata": {}, "outputs": [], "source": [ @@ -1678,48 +1594,47 @@ }, { "cell_type": "code", - "execution_count": 49, + "execution_count": 53, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "[{'score': 15.986347, 'text': 'Denver Broncos'},\n", - " {'score': 14.585561,\n", + "[{'score': 15.317053, 'text': 'Denver Broncos'},\n", + " {'score': 13.32964,\n", " 'text': 'Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers'},\n", - " {'score': 13.152991, 'text': 'Carolina Panthers'},\n", - " {'score': 12.38233, 'text': 'Broncos'},\n", - " {'score': 10.981544,\n", - " 'text': 'Broncos defeated the National Football Conference (NFC) champion Carolina Panthers'},\n", - " {'score': 10.852013,\n", + " {'score': 11.520305, 'text': 'Carolina Panthers'},\n", + " {'score': 11.3409395, 'text': 'Broncos'},\n", + " {'score': 9.816772,\n", " 'text': 'American Football Conference (AFC) champion Denver Broncos'},\n", - " {'score': 10.635618,\n", + " {'score': 9.63557, 'text': 'Denver'},\n", + " {'score': 9.372208,\n", " 'text': 'The American Football Conference (AFC) champion Denver Broncos'},\n", - " {'score': 10.283654, 'text': 'Denver'},\n", - " {'score': 9.451225,\n", + " {'score': 9.353527,\n", + " 'text': 'Broncos defeated the National Football Conference (NFC) champion Carolina Panthers'},\n", + " {'score': 8.184068,\n", + " 'text': 'Denver Broncos defeated the National Football Conference'},\n", + " {'score': 7.829359,\n", " 'text': 'American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers'},\n", - " {'score': 9.234833,\n", + " {'score': 7.441773, 'text': 'champion Denver Broncos'},\n", + " {'score': 7.384794,\n", " 'text': 'The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers'},\n", - " {'score': 8.7582445,\n", - " 'text': 'Denver Broncos defeated the National Football Conference'},\n", - " {'score': 8.187819,\n", - " 'text': 'Denver Broncos defeated the National Football Conference (NFC) champion Carolina'},\n", - " {'score': 8.134832, 'text': 'Panthers'},\n", - " {'score': 8.092252,\n", + " {'score': 7.098033,\n", " 'text': 'Denver Broncos defeated the National Football Conference (NFC)'},\n", - " {'score': 7.7162285,\n", - " 'text': 'the National Football Conference (NFC) champion Carolina Panthers'},\n", - " {'score': 7.595868,\n", - " 'text': 'Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10'},\n", - " {'score': 7.382572,\n", - " 'text': 'National Football Conference (NFC) champion Carolina Panthers'},\n", - " {'score': 7.320059,\n", + " {'score': 6.7240176,\n", + " 'text': 'Denver Broncos defeated the National Football Conference (NFC) champion Carolina'},\n", + " {'score': 6.4590645,\n", " 'text': 'Denver Broncos defeated the National Football Conference (NFC'},\n", - " {'score': 6.755249, 'text': 'Carolina'},\n", - " {'score': 6.728976, 'text': 'champion Denver Broncos'}]" + " {'score': 6.4571347, 'text': 'AFC) champion Denver Broncos'},\n", + " {'score': 6.124057, 'text': 'Panthers'},\n", + " {'score': 5.7547617,\n", + " 'text': 'Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10'},\n", + " {'score': 5.45436,\n", + " 'text': 'champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers'},\n", + " {'score': 4.914682, 'text': 'Carolina'}]" ] }, - "execution_count": 49, + "execution_count": 53, "metadata": {}, "output_type": "execute_result" } @@ -1772,7 +1687,7 @@ }, { "cell_type": "code", - "execution_count": 50, + "execution_count": 59, "metadata": {}, "outputs": [ { @@ -1782,7 +1697,7 @@ " 'answer_start': [177, 177, 177]}" ] }, - "execution_count": 50, + "execution_count": 59, "metadata": {}, "output_type": "execute_result" } @@ -1808,7 +1723,7 @@ }, { "cell_type": "code", - "execution_count": 51, + "execution_count": 60, "metadata": {}, "outputs": [], "source": [ @@ -1836,7 +1751,7 @@ }, { "cell_type": "code", - "execution_count": 52, + "execution_count": 61, "metadata": {}, "outputs": [], "source": [ @@ -1930,7 +1845,7 @@ }, { "cell_type": "code", - "execution_count": 53, + "execution_count": 62, "metadata": {}, "outputs": [ { @@ -1943,7 +1858,7 @@ { "data": { "application/vnd.jupyter.widget-view+json": { - "model_id": "b9fcc5ebf7d246008c7ec68f5806182c", + "model_id": "d732a04a2dda436a88b2744ac7728d62", "version_major": 2, "version_minor": 0 }, @@ -1968,16 +1883,16 @@ }, { "cell_type": "code", - "execution_count": 54, + "execution_count": 63, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "/tmp/ipykernel_20254/2330875496.py:3: FutureWarning: load_metric is deprecated and will be removed in the next major version of datasets. Use 'evaluate.load' instead, from the new library 🤗 Evaluate: https://huggingface.co/docs/evaluate\n", + "/tmp/ipykernel_2152/2330875496.py:3: FutureWarning: load_metric is deprecated and will be removed in the next major version of datasets. Use 'evaluate.load' instead, from the new library 🤗 Evaluate: https://huggingface.co/docs/evaluate\n", " metric = load_metric(\"squad_v2\" if squad_v2 else \"squad\")\n", - "/root/miniconda3/lib/python3.11/site-packages/datasets/load.py:752: FutureWarning: The repository for squad contains custom code which must be executed to correctly load the metric. You can inspect the repository content at https://raw.githubusercontent.com/huggingface/datasets/2.16.1/metrics/squad/squad.py\n", + "/root/miniconda3/envs/myenv/lib/python3.11/site-packages/datasets/load.py:752: FutureWarning: The repository for squad contains custom code which must be executed to correctly load the metric. You can inspect the repository content at https://raw.githubusercontent.com/huggingface/datasets/2.16.1/metrics/squad/squad.py\n", "You can avoid this message in future by passing the argument `trust_remote_code=True`.\n", "Passing `trust_remote_code=True` will be mandatory to load this metric from the next major release of `datasets`.\n", " warnings.warn(\n" @@ -1986,7 +1901,7 @@ { "data": { "application/vnd.jupyter.widget-view+json": { - "model_id": "c528a0f9cb2c4787b5c6ee2342fa1519", + "model_id": "98a871ae556a43508aae6627bbc14336", "version_major": 2, "version_minor": 0 }, @@ -2000,7 +1915,7 @@ { "data": { "application/vnd.jupyter.widget-view+json": { - "model_id": "5f681da0b712486fad8521f4a7d532a8", + "model_id": "40830a571c104cf9965dcda422dca0fe", "version_major": 2, "version_minor": 0 }, @@ -2031,16 +1946,16 @@ }, { "cell_type": "code", - "execution_count": 55, + "execution_count": 64, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "{'exact_match': 74.88174077578051, 'f1': 83.6359321422016}" + "{'exact_match': 74.63576158940397, 'f1': 83.46282700326206}" ] }, - "execution_count": 55, + "execution_count": 64, "metadata": {}, "output_type": "execute_result" } @@ -2070,23 +1985,51 @@ }, { "cell_type": "code", - "execution_count": 50, + "execution_count": 18, "metadata": {}, "outputs": [], "source": [ - "trained_model = AutoModelForQuestionAnswering.from_pretrained(model_dir)" + "from transformers import AutoModelForQuestionAnswering, TrainingArguments, Trainer\n", + "batch_size=64\n", + "model_checkpoint = \"distilbert-base-uncased\"\n", + "model_dir = f\"models/{model_checkpoint}-finetuned-squad\"\n", + "trained_model = AutoModelForQuestionAnswering.from_pretrained(model_dir)\n", + "args = TrainingArguments(\n", + " output_dir=model_dir,\n", + " evaluation_strategy = \"epoch\",\n", + " learning_rate=2e-5,\n", + " per_device_train_batch_size=batch_size,\n", + " per_device_eval_batch_size=batch_size,\n", + " num_train_epochs=3,\n", + " weight_decay=0.01,\n", + " save_strategy='epoch', # 'no', 'epoch', 'steps' - 控制保存策略\n", + " save_steps=500, # 如果save_strategy='steps'，每多少步保存一次\n", + " save_total_limit=5, # 只保存最近的几个检查点\n", + " load_best_model_at_end=True, # 训练结束时加载最好的模型\n", + ")" ] }, { "cell_type": "code", - "execution_count": 51, + "execution_count": 19, + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import default_data_collator\n", + "\n", + "data_collator = default_data_collator" + ] + }, + { + "cell_type": "code", + "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "Detected kernel version 4.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n" + "Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n" ] } ], @@ -2101,12 +2044,578 @@ ")" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "

\n", + " \n", + " \n", + " [2221/4152 1:15:30 < 1:05:42, 0.49 it/s, Epoch 1.60/3]\n", + "
\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
EpochTraining LossValidation Loss
10.8570001.218290

" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Checkpoint destination directory models/distilbert-base-uncased-finetuned-squad/checkpoint-1384 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n" + ] + } + ], + "source": [ + "trained_trainer.train()" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AutoGPTQ_transformers.ipynb\t fine-tune-quickstart.ipynb\n", + "AWQ_transformers.ipynb\t\t models\n", + "AWQ_transformers-opt-2.7b.ipynb pipelines_advanced.ipynb\n", + "data\t\t\t\t pipelines.ipynb\n", + "docs\t\t\t\t quantization_bnb.ipynb\n", + "fine-tune-QA.ipynb\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", + "To disable this warning, you can either:\n", + "\t- Avoid using `tokenizers` before the fork if possible\n", + "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n" + ] + } + ], + "source": [ + "!ls" + ] + }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], - "source": [] + "source": [ + "!ls -l models/distilbert-base-uncased-finetuned-squad/checkpoint-4152" + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "

\n", + " \n", + " \n", + " [169/169 01:48]\n", + "
\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'eval_loss': 1.2182899713516235, 'eval_runtime': 108.814, 'eval_samples_per_second': 99.105, 'eval_steps_per_second': 1.553, 'epoch': 3.0}\n" + ] + } + ], + "source": [ + "eval_results = trained_trainer.evaluate()\n", + "print(eval_results)" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " [4152/4152 : < :, Epoch 3/3]\n", + "
\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
EpochTraining LossValidation Loss

" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/plain": [ + "TrainOutput(global_step=4152, training_loss=0.0, metrics={'train_runtime': 0.1204, 'train_samples_per_second': 2206373.171, 'train_steps_per_second': 34494.832, 'total_flos': 2.602335381127373e+16, 'train_loss': 0.0, 'epoch': 3.0})" + ] + }, + "execution_count": 34, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "trained_trainer.train(resume_from_checkpoint=\"models/distilbert-base-uncased-finetuned-squad/checkpoint-4152\")" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [], + "source": [ + "model_to_save = trained_trainer.save_model(model_dir)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "评估重新训练的模型：" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "odict_keys(['loss', 'start_logits', 'end_logits'])" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import torch\n", + "\n", + "for batch in trained_trainer.get_eval_dataloader():\n", + " break\n", + "batch = {k: v.to(trained_trainer.args.device) for k, v in batch.items()}\n", + "with torch.no_grad():\n", + " output = trained_trainer.model(**batch)\n", + "output.keys()" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(torch.Size([64, 384]), torch.Size([64, 384]))" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "output.start_logits.shape, output.end_logits.shape" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(tensor([ 46, 57, 78, 43, 118, 15, 72, 35, 15, 34, 73, 41, 80, 91,\n", + " 156, 35, 83, 91, 80, 58, 77, 31, 42, 53, 41, 35, 42, 77,\n", + " 11, 44, 27, 133, 66, 40, 87, 44, 43, 83, 127, 26, 28, 33,\n", + " 87, 127, 95, 25, 43, 132, 42, 29, 44, 46, 24, 44, 65, 58,\n", + " 81, 14, 59, 72, 25, 36, 57, 43], device='cuda:0'),\n", + " tensor([ 47, 58, 81, 44, 118, 110, 75, 37, 110, 36, 76, 42, 83, 94,\n", + " 158, 35, 83, 94, 83, 60, 80, 74, 43, 54, 42, 35, 43, 80,\n", + " 13, 45, 28, 133, 66, 41, 89, 45, 87, 85, 127, 27, 30, 34,\n", + " 89, 127, 97, 26, 44, 132, 43, 30, 45, 47, 25, 45, 65, 59,\n", + " 81, 14, 60, 72, 25, 36, 58, 43], device='cuda:0'))" + ] + }, + "execution_count": 25, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "output.start_logits.argmax(dim=-1), output.end_logits.argmax(dim=-1)" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [], + "source": [ + "n_best_size = 20" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "\n", + "start_logits = output.start_logits[0].cpu().numpy()\n", + "end_logits = output.end_logits[0].cpu().numpy()\n", + "\n", + "# 获取最佳的起始和结束位置的索引：\n", + "start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n", + "end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n", + "\n", + "valid_answers = []\n", + "\n", + "# 遍历起始位置和结束位置的索引组合\n", + "for start_index in start_indexes:\n", + " for end_index in end_indexes:\n", + " if start_index <= end_index: # 需要进一步测试以检查答案是否在上下文中\n", + " valid_answers.append(\n", + " {\n", + " \"score\": start_logits[start_index] + end_logits[end_index],\n", + " \"text\": \"\" # 我们需要找到一种方法来获取与上下文中答案对应的原始子字符串\n", + " }\n", + " )" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "metadata": {}, + "outputs": [], + "source": [ + "def prepare_validation_features(examples):\n", + " # 一些问题的左侧有很多空白，这些空白并不有用且会导致上下文截断失败（分词后的问题会占用很多空间）。\n", + " # 因此我们移除这些左侧空白\n", + " examples[\"question\"] = [q.lstrip() for q in examples[\"question\"]]\n", + "\n", + " # 使用截断和可能的填充对我们的示例进行分词，但使用步长保留溢出的令牌。这导致一个长上下文的示例可能产生\n", + " # 几个特征，每个特征的上下文都会稍微与前一个特征的上下文重叠。\n", + " tokenized_examples = tokenizer(\n", + " examples[\"question\" if pad_on_right else \"context\"],\n", + " examples[\"context\" if pad_on_right else \"question\"],\n", + " truncation=\"only_second\" if pad_on_right else \"only_first\",\n", + " max_length=max_length,\n", + " stride=doc_stride,\n", + " return_overflowing_tokens=True,\n", + " return_offsets_mapping=True,\n", + " padding=\"max_length\",\n", + " )\n", + "\n", + " # 由于一个示例在上下文很长时可能会产生几个特征，我们需要一个从特征映射到其对应示例的映射。这个键就是为了这个目的。\n", + " sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n", + "\n", + " # 我们保留产生这个特征的示例ID，并且会存储偏移映射。\n", + " tokenized_examples[\"example_id\"] = []\n", + "\n", + " for i in range(len(tokenized_examples[\"input_ids\"])):\n", + " # 获取与该示例对应的序列（以了解哪些是上下文，哪些是问题）。\n", + " sequence_ids = tokenized_examples.sequence_ids(i)\n", + " context_index = 1 if pad_on_right else 0\n", + "\n", + " # 一个示例可以产生几个文本段，这里是包含该文本段的示例的索引。\n", + " sample_index = sample_mapping[i]\n", + " tokenized_examples[\"example_id\"].append(examples[\"id\"][sample_index])\n", + "\n", + " # 将不属于上下文的偏移映射设置为None，以便容易确定一个令牌位置是否属于上下文。\n", + " tokenized_examples[\"offset_mapping\"][i] = [\n", + " (o if sequence_ids[k] == context_index else None)\n", + " for k, o in enumerate(tokenized_examples[\"offset_mapping\"][i])\n", + " ]\n", + "\n", + " return tokenized_examples" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "metadata": {}, + "outputs": [], + "source": [ + "validation_features = datasets[\"validation\"].map(\n", + " prepare_validation_features,\n", + " batched=True,\n", + " remove_columns=datasets[\"validation\"].column_names\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "raw_predictions = trained_trainer.predict(validation_features)" + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "metadata": {}, + "outputs": [], + "source": [ + "validation_features.set_format(type=validation_features.format[\"type\"], columns=list(validation_features.features.keys()))" + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "metadata": {}, + "outputs": [], + "source": [ + "from tqdm.auto import tqdm\n", + "import collections\n", + "import numpy as np\n", + "\n", + "def postprocess_qa_predictions(examples, features, raw_predictions, n_best_size = 20, max_answer_length = 30):\n", + " all_start_logits, all_end_logits = raw_predictions\n", + " # 构建一个从示例到其对应特征的映射。\n", + " example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n", + " features_per_example = collections.defaultdict(list)\n", + " for i, feature in enumerate(features):\n", + " features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)\n", + "\n", + " # 我们需要填充的字典。\n", + " predictions = collections.OrderedDict()\n", + "\n", + " # 日志记录。\n", + " print(f\"正在后处理 {len(examples)} 个示例的预测，这些预测分散在 {len(features)} 个特征中。\")\n", + "\n", + " # 遍历所有示例！\n", + " for example_index, example in enumerate(tqdm(examples)):\n", + " # 这些是与当前示例关联的特征的索引。\n", + " feature_indices = features_per_example[example_index]\n", + "\n", + " min_null_score = None # 仅在squad_v2为True时使用。\n", + " valid_answers = []\n", + " \n", + " context = example[\"context\"]\n", + " # 遍历与当前示例关联的所有特征。\n", + " for feature_index in feature_indices:\n", + " # 我们获取模型对这个特征的预测。\n", + " start_logits = all_start_logits[feature_index]\n", + " end_logits = all_end_logits[feature_index]\n", + " # 这将允许我们将logits中的某些位置映射到原始上下文中的文本跨度。\n", + " offset_mapping = features[feature_index][\"offset_mapping\"]\n", + "\n", + " # 更新最小空预测。\n", + " cls_index = features[feature_index][\"input_ids\"].index(tokenizer.cls_token_id)\n", + " feature_null_score = start_logits[cls_index] + end_logits[cls_index]\n", + " if min_null_score is None or min_null_score < feature_null_score:\n", + " min_null_score = feature_null_score\n", + "\n", + " # 浏览所有的最佳开始和结束logits，为 `n_best_size` 个最佳选择。\n", + " start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n", + " end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n", + " for start_index in start_indexes:\n", + " for end_index in end_indexes:\n", + " # 不考虑超出范围的答案，原因是索引超出范围或对应于输入ID的部分不在上下文中。\n", + " if (\n", + " start_index >= len(offset_mapping)\n", + " or end_index >= len(offset_mapping)\n", + " or offset_mapping[start_index] is None\n", + " or offset_mapping[end_index] is None\n", + " ):\n", + " continue\n", + " # 不考虑长度小于0或大于max_answer_length的答案。\n", + " if end_index < start_index or end_index - start_index + 1 > max_answer_length:\n", + " continue\n", + "\n", + " start_char = offset_mapping[start_index][0]\n", + " end_char = offset_mapping[end_index][1]\n", + " valid_answers.append(\n", + " {\n", + " \"score\": start_logits[start_index] + end_logits[end_index],\n", + " \"text\": context[start_char: end_char]\n", + " }\n", + " )\n", + " \n", + " if len(valid_answers) > 0:\n", + " best_answer = sorted(valid_answers, key=lambda x: x[\"score\"], reverse=True)[0]\n", + " else:\n", + " # 在极少数情况下我们没有一个非空预测，我们创建一个假预测以避免失败。\n", + " best_answer = {\"text\": \"\", \"score\": 0.0}\n", + " \n", + " # 选择我们的最终答案：最佳答案或空答案（仅适用于squad_v2）\n", + " if not squad_v2:\n", + " predictions[example[\"id\"]] = best_answer[\"text\"]\n", + " else:\n", + " answer = best_answer[\"text\"] if best_answer[\"score\"] > min_null_score else \"\"\n", + " predictions[example[\"id\"]] = answer\n", + "\n", + " return predictions" + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "正在后处理 10570 个示例的预测，这些预测分散在 10784 个特征中。\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "337dd1b8a08b4fae94cb26c3dde760eb", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + " 0%| | 0/10570 [00:00\n", " \n", " 0\n", - " 2 star\n", - " I stopped in for dinner last night, craving their grits... the only thing that was any good during this visit were the fried eggs. I ordered the High Flyer again, my usual order at both locations, and it was a wreck. \\n\\nService was friendly and attentive, as usual. On a Friday night, there were two other tables seated, and they were finishing up when I sat down. Both Biscuits refuse to fully air condition the restaurant, so it was as tropical as the table cloths. \\n\\nThe food:\\nThe eggs, cooked to over-well, were perfect. \\nThe chicken sausage was over cooked and almost crunchy. So overly seasoned that I woke up at 1 am and it felt like my chest was on fire from indigestion.\\nThe biscuit - well, I stated before how I feel about their biscuits. Too heavy and really just a way to get the delicious apple cranberry butter into my pie hole. But at least it wasn't burned.\\nI substituted a buttermilk pancake for the oatmeal pancake and it was burned AGAIN. It's a pancake - if you burn it, throw it out and give me another one - it's not a steak, it's batter that probably costs less than a nickel. \\nThe grits - my favorite grits - grits that make my mouth water every time I drive by this or the Park Rd location - were AWFUL. Overly cheesed and overly salty, clumps of grits were stuck together - BAD. \\n\\nAs my Pawpaw said, during his final days, after spending the weekend with a family member that will make you wish death would come soon, \\\"I ain't goin' back\\\".\n", + " 3 stars\n", + " If you go to Cave Creek you must go to the Buffalo Chip, Harold's , hideaway etc. just great local restaurants in a great town north of Phoenix/Scottsdale. I like the Chip more for drinking dancing and rodeo than the food. But it is okay for a little grub But it is fun. So try it\n", " \n", " \n", " 1\n", - " 2 star\n", - " This was my first visit to this location for Sunday Champagne Brunch. It was located in the Cannery Casino on E Craig Road in Las Vegas. We had free buffet coupons from spinning the wheel on Web Pass.\\n\\nThe stations were not labeled, so you had to walk up to see what there was. At the Cannery Row Buffet on Boulder Highway, the stations were clearly marked with big signs. They had American, Asian, Italian, Mexican, 2 large Salad Bars, and Desserts. The server took our drink order and brought us drinks and cleared our empty plates. We were not offered champagne.\\n\\nFor brunch there was a made to order omlette station. This was probably the best and freshest feature. There was no Egg Benedict. The bacon was limp and greasy. The hash browns were shredded and burned. The sausage links had a lot of filler. The gravy for the Biscuits and Gravy had been sitting there for a while.\\n\\nThe Italian meatballs were large, but had a lot of filler. The tortellini was meh.\\n\\nThe taquitos were ok if you like the kind that come in the freezer section. The cheese sauce was runny. There were tamales wrapped in plastic wrap that I did not try. \\n\\nThe potstickers were dried out and fried. The Broccoli Beef was okay, but had thin strips of beef. The egg rolls were tastless.\\n\\nThe pepperoni pizza was okay when it was fresh and hot. There was cheese bread.\\n\\nThe salad bar had lettuce, lots of toppings, pasta salad, ambrosia, assorted fruit, rolls, etc. This was much nicer than the Cannery Row Buffet on Boulder Highway.\\n\\nAgain, the desserts were the best part of the meal. I had to ask for the Coconut Cream Pie. Yumm! The Apple Custard Tart was good (I had 2 slices). There was Boston Cream Pie, Carrot Cake, Cheesecake, Tiramisu, wedding cookies, soft-serve ice cream, and plenty of sugar-free desserts.\n", + " 3 stars\n", + " Chuy's was pretty good. Seems like all of their locations are pretty similar. I think they'd be best known for their cheap margaritas... $2 pints, $4 small pitchers. They aren't too strong and come from a mediocre mix, but if you're looking for something sweet and cheap its a good bet.\\n\\nFree serve-yourself chips and salsa are pretty good. \\n\\nAs for food... pretty decent prices and okay to good food. I wouldn't say its authentic Mexican, actually I think the menu is a bit confused. But the food is good overall. It's a place we'll go to every month or so.\n", " \n", " \n", " 2\n", - " 1 star\n", - " The road is paved now so that's great news. Unfortunately it's very over priced. When you finally get there, you still have to get on another shuttle bus to get to the skywalk. The staff was VERY unprofessional. You are not allowed to take your own camera so they can charge you a crazy amount to take pictures themselves. While sightseeing, we were constantly being asked to move so the staff could take pictures of other tourists. They try to usher you off the Skywalk so they can get in the next group. \\n\\nGreat views but terrible service. Not worth the time, effort or price. I would like 2/3 of my money back please.\n", + " 3 stars\n", + " I ended up eating at Taggia while staying at Firesky so it was a choice of convenience. I've had the food from here several times using room service and it's never anything to complain about. It was the same story the day I had lunch here. I had an organic greens salad and shared the margherita and goat cheese pizzas with my fellow lunchers. All of the food was good - the goat cheese pizza in particular with its thin, crispy crust.\\n\\nUnfortunately the day we ate here our service was MIA. We were told we could seat ourselves so we did. After about 10 minutes someone came by to take our drink order and maybe 10 minutes later our waters arrived. Well 2 out of 3 of them did anyway. Then we ordered two salads and two pizzas to share. One pizza came first. WTH? Where were the salads? Or the other pizza? The salads showed up a few minutes later and then our server realized that she had forgotten our second pizza. No biggie since we had salads and one pizza to eat. But the service was lackluster with a L. Like Andrea R says, I wouldn't go out of my way to eat here, but when in the area it's a good option to have.\n", " \n", " \n", " 3\n", - " 4 stars\n", - " One of the best pizzas I've had in Charlotte. They also sell pizza by the slice which is a plus.\n", + " 2 star\n", + " I recently had a work luncheon at Ricardo's, I had been before years ago and it was extremely unmemorable. This visit would be more memorable but for the wrong reasons. \\n\\nWhen given the choice, I prefer to order off the menu than choose a buffet. But the whole group went to the buffet and I didn't want to be the oddball. I had two carne asada tacos, cheese enchilada and chips & salsa. The enchilada was bland the only hint of flavor was the acidity from the tomatoes. The salsa, too, was bland and watery. The chips were pretty generic. The first taco was ok, a bit bland, but tender. The second was filled with grizzly meat. It really turned my stomach. Fortunately, the service was friendly and they were able to accomodate our large group.\n", " \n", " \n", " 4\n", - " 2 star\n", - " I try cupcake shops everywhere I go and was excited to stop here. They're located inside a fantastic building with other quaint little shops surrounding it. We picked up a dozen minis ( that's all they sell at this location) of pre selected pre packaged flavors. There were 2 each of six different flavors cupcakes in the package .... But it might as well have been 3 flavors! Unless you have a chocolate or red Velvet all the cake is the same! Boringgggggggg! The Snickerdoodle, birthday and vanilla all taste the same the only difference is the sprinkles on top. Such a disappointment . I will say they were very moist which is why I gave 2 stars instead of 1.\n", + " 4 stars\n", + " We had a great time at this resort over the long weekend. The staff was super friendly, especially Adam, David and Cassie. Great job!!! And our suite was perfect to accommodate three women with lots of bags, make-up and shoes. The Hole in the Wall restaurant had a really good breakfast, friendly staff and an outdoors patio. Not so for the Rico Restaurant. They were a bit rude, overwhelmed and obviously didn't want our business. We also floated down the Lazy River, it was definitely Lazy...pretty slow but perfect temp. All in all, I'll be back.\n", " \n", " \n", " 5\n", - " 2 star\n", - " So first off, an orange table on a sign on a pillar facing the opposite direction of the entrance into the park from the parking garage is not a great way to show people that your business is in fact right there. We wandered to the left and right of the prominent Jewel of the Crown and finally had to call the brusque busy man who half thought we were at the Fashion Square though I started the conversation with where exactly in the mall are you because we're in front of AZ88. \\n\\nWe finally figure out the door which was also covered by a huge 7 foot tall man watching another man on a ladder screw in a bracket. Both men have their backs to us and are completely blocking the door. We walk up behind them and watch the man screw in the bracket and finally the tall man moves aside just enough for me to get to the door. Guess they weren't expecting customers.\\n\\nThe food itself isn't awful. The bloody mary is tiny for a $7 cocktail and both burger and sandwich were a bit overpriced for what they were. They both came with some potato chips and a pickle and came in a little over $10 each. The burger was actually decent, though it was a little rare for a medium. The flavor of the meat was actually quite nice. The sandwich was ok, though it almost seemed like sliced deli meat pan fried (not a breast). \\n\\nThey're shutting down and renewing themselves as some other establishment. Hopefully they'll have a sign up by then.\n", + " 1 star\n", + " Im an owner with no kids, this place is not for my husband and I.. The element here is all about families and cooking in and playing in the pool from the moment it opens.\\n\\nThe restaurant bar is a bit of a joke, and the pressure to buy more points makes a relaxing vacation more stressful. We were an original owner and saw most of it built.\\n\\nWe noticed that they no longer offer a shuttle which is a mistake for those that want to go to the strip and not have to worry about driving. But after this weekend I see that they don't need to offer the shuttle because more than half the people there don't plan on leaving the facility at all.\\n\\nThe guests that we ran into all seemed to be there on a free vacation offer. They were tatted up and pushing baby strollers... and screaming to each other WAIT TIL YOU SEE THE POOL....\\n\\nMy hubby and myself both looked at each other and said OUT LOUD, we don't think we will be coming back here again ever to this location.\\n\\nWe came home and looked into selling it all together, but then thought maybe we would try another location that Diamond Resorts has to offer before we do so..\\n\\nSo bottom line, if you have kids and love the pool and slides and pool some more.. this is for you.. If your looking for a weekend with the hubby or friends in Vegas to relax and to enjoy what Vegas is all about... this resort is not for you..\n", " \n", " \n", " 6\n", - " 5 stars\n", - " YAY!! Rachel C at EWC is awesome, thank you sooooo much for her. I \\\"go to brazil,\\\" with Rachel every 3-4 weeks. She has always been on time, very nice, and professional. She makes me feel comfortable, and is very cean, never a hair missed. :)\n", + " 3 stars\n", + " Booked a room here through Priceline for the Tuesday before Thanksgiving. Actually booked it on the drive in from Las Vegas through my cell phone, which was pretty sweet. Paid $25 + tax, so you can't beat the price. We had a hard time finding it as Google Maps was wrong about it's location, but you can't blame the hotel for that.\\n\\nWhat I can blame the hotel for is not giving me a king size bed. Priceline had booked me with 2 doubles, and in my experience I am always able to switch unless they're sold out. The front desk clerk told me they were indeed not sold out, but it was their policy not to let Priceline users switch rooms. So much for considering staying there the rest of Thanksgiving week.\\n\\nI was going to let it go, but then at 9:30am the maid knocked on our door and woke me up 2 hours earlier than I had planned (we got to sleep at 5am, give me a break). I ended up coming out of my room and saw that my do not disturb sign was on, so she must have chosen to ruin my day for fun. Tried to fall back asleep but was then kept up by the sounds of what looked to be a loud garbage truck parked right outside of our room.\\n\\nI give up on sleeping. At least they have solid free Internet so I can Yelp this hotel. Courtyard, you're lucky you're getting 3 stars from me.\n", " \n", " \n", " 7\n", - " 3 stars\n", - " Some good sights seeing while at the stadium. Good view from where ever you sit. Had a great time inside and out during the recent Sevens Rugby tournament festivities. But it is old and outdated. OK at best. They should level it and put just off the strip.\n", + " 2 star\n", + " Been to 4 Cirques and this is the least favorite. Sets and costumes are absolutely amazing but the acts were very unimpressive compared to the older ones we've seen. \\nThe only exception was the opening act with the two twin men on ropes that swung out into the audience. They weren't in the book at the shop so I think they added them in later to spice up the program. Breathtaking!\\n\\nPros- Art direction, sets, lights\\n\\nCons- Acts seen before and ANNOYING clowns\n", " \n", " \n", " 8\n", - " 1 star\n", - " Shared a room with a friend who flew into town for business and didn't want to stay alone. I was confused as to why she would chose the Artisan but her company arranged all her travel plans. She asked for my company for the night after reading some reviews.\\n\\nThe one star is for the eclectic and artsy decor, which I love, because it was a refreshing change from what we normally fund in Vegas. The staff was pretty cool too.\\n\\nBUT.... I will never book my own stay here or come to hang at the bar. The drinks were not good and way over priced. The carpet in my friend's suite was so dirty the bottom of my yellow flip flops turned black, only I didn't realize this until I checked them because I had taken them off and the carpet felt sticky on my bare feet. There was an ashtray sitting outside the window, which was full, and trash strewn across the roof was our view. Bathroom fixtures were broken and the radio/dvd player didn't work.\\n\\nEVEN WORSE was the fact that we came out of the room TWICE to a creeper with his ear plastered to our door. What was supposed to be a chill evening evening turned into us escaping to the Las Vegas strip to relax.\n", + " 2 star\n", + " KOOLAID KID reminded me of home...nice touch..i know I know..this is suppose to be about the chicken & waffles, but I must say quenching my thirst is very important to me..so back to the food..it was just that chicken(no flavor) & waffles(nothing special)..mac & cheese was very nice...and the new building was very very nice..okay that's all\n", " \n", " \n", " 9\n", - " 4 stars\n", - " On a perfectly mild sunny day in vegas, it was a great choice to eat on the patio at Mon Ami Gabi overlooking Las Vegas Blvd. As soon as we entered the restaurant from the casino entrance, from the dark interior, it was like stepping into a whole different world once we got seated on the patio. \\nUsually when servers rattle off the specials of the day, it goes out the other ear for me. But after looking over the menu and half heartedly deciding on the spinach and gruyere omlette, I heard the server mention Bacon and Spinach Eggs Benedict and i was sold! Quiche Lorraine with bacon http://www.yelp.com/biz_photos/4JNXUYY8wbaaDmk3BPzlWw?select=4Ua39HREZlvlYbEf3EP_ew for the dude and my friend got the dish i was thinking of originally http://www.yelp.com/biz_photos/4JNXUYY8wbaaDmk3BPzlWw?select=5luau0SKUmjSVfXBalxsMQ. We all agreed that my dish ended up being the best: http://www.yelp.com/biz_photos/4JNXUYY8wbaaDmk3BPzlWw?select=T3jPSmE56xcETgTqjegttw.\\nThe quiche had a nice crust, but was kind of monotonous. (i love breakfast sides and this had none). The spinach omelette was ok- i enjoyed my omelette at Max Brenner much more. \\nI've only had eggs benedict a few times elsewhere and i have to say this one was very yummy- the eggs were perfectly runny and the bacon (i believe it was applewood) was nice and crispy. \\nWe were too sweets wasted this time to order the pancakes, french toast etc, so next time i would like to try those or the steak dishes with a side of their onion soup!\n", + " 1 star\n", + " Just called this location and I live 1.8 miles away. I asked them to deliver and they informed me that they would not deliver to my house because it was a couple hundred yards out of the map plan. They asked me to call the power and southern store. This store advised me that they could not deliver because jimmy johns has a two mile radius they can deliver to. Called this store back and they once again decided to tell me even though I was in the two mile radius they did not want to deliver to me and my only option was for pickup. I will never eat at this location. I know the owners at Firehouse Subs and they go out of the way and this location is just lazy. Not getting my money jimmy johns no matter how fast you are. Laziness is worse\n", " \n", " \n", "" @@ -261,10 +332,95 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 8, "id": "8bf2b342-e1dd-4ab6-ad57-28eb2513ae38", "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "375d0c0ccbd546629ed82bbfbf3daab9", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "tokenizer_config.json: 0%| | 0.00/29.0 [00:00\n", " \n", " 0\n", - " 2 star\n", - " Here's the deal: Long Lines - Overpriced - Good Service - Average Food.\\n\\nI would LOVE IT if Steve would actually eat in his own buffet so he could see what it has become. When we were here years past the quality of the food was so much better (but the service was lacking). Now it's the opposite-- the service is good but you can really notice the decline in the food (cuts of meat, etc). And, two small glasses of wine will set you back a good $25. \\n\\nI know it's the strip and I don't mind paying $40 a person IF the food is good (you'd pay that in a decent restaurant anywhere else). My problem is that the Wynn prides itself on quality... \\n\\nActually, to be honest I've decided I now officially hate buffets. I can't stand watching people stuffing themselves and coming back to their table with plates piled high with 50+ crab legs. Not to mention the amount of food that is wasted is ridiculous. \\n\\nPeople, what have we become? \\n\\nBTW, if you want a good buffet, skip the Wynn and head over to the new one at the Cosmopolitan Hotel.\n", - " [101, 3446, 112, 188, 1103, 2239, 131, 3261, 12058, 118, 3278, 1643, 10835, 1181, 118, 2750, 2516, 118, 18098, 6702, 119, 165, 183, 165, 183, 2240, 1156, 149, 2346, 17145, 9686, 1191, 3036, 1156, 2140, 3940, 1107, 1117, 1319, 171, 9435, 2105, 1177, 1119, 1180, 1267, 1184, 1122, 1144, 1561, 119, 1332, 1195, 1127, 1303, 1201, 1763, 1103, 3068, 1104, 1103, 2094, 1108, 1177, 1277, 1618, 113, 1133, 1103, 1555, 1108, 11744, 114, 119, 1986, 1122, 112, 188, 1103, 3714, 118, 118, 1103, 1555, 1110, 1363, 1133, 1128, 1169, 1541, 4430, 1103, 6246, 1107, 1103, 2094, 113, 7484, 1104, 6092, ...]\n", + " 5 stars\n", + " What a place! It was my first time at Aria and I simply loved it! We decided to eat at Javier's since the decor was so inviting. You have to see it to believe it. The food is super fresh and extremely tasty and the service is top notch. \\n\\nI wasn't feeling too hungry so I got the ceviche with octopus on recommendation from our server and it was probably the best I have ever eaten. The chips are fresh and the salsa is spicy!\\n\\nThe design detail of this place is outstanding. You have to go there to take a look even if you're not close by, it's well worth the cab ride. I hope to go back and have a full meal.\n", + " [101, 1327, 170, 1282, 106, 1135, 1108, 1139, 1148, 1159, 1120, 12900, 1105, 146, 2566, 3097, 1122, 106, 1284, 1879, 1106, 3940, 1120, 14280, 112, 188, 1290, 1103, 1260, 19248, 1108, 1177, 16067, 119, 1192, 1138, 1106, 1267, 1122, 1106, 2059, 1122, 119, 1109, 2094, 1110, 7688, 4489, 1105, 4450, 27629, 13913, 1105, 1103, 1555, 1110, 1499, 23555, 119, 165, 183, 165, 183, 2240, 1445, 112, 189, 2296, 1315, 7555, 1177, 146, 1400, 1103, 172, 21075, 1162, 1114, 184, 5822, 17298, 1113, 13710, 1121, 1412, 9770, 1105, 1122, 1108, 1930, 1103, 1436, 146, 1138, 1518, 8527, 119, 1109, 13228, 1132, ...]\n", " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...]\n", " [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...]\n", " \n", @@ -316,10 +472,21 @@ }, "metadata": {}, "output_type": "display_data" + }, + { + "data": { + "text/plain": [ + "650000" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" } ], "source": [ - "show_random_elements(tokenized_datasets[\"train\"], num_examples=1)" + "show_random_elements(tokenized_datasets[\"train\"], num_examples=1)\n", + "tokenized_datasets[\"train\"].num_rows" ] }, { @@ -336,7 +503,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 14, "id": "a17317d8-3c6a-467f-843d-87491f600db1", "metadata": {}, "outputs": [], @@ -359,15 +526,32 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 15, "id": "4d2af4df-abd4-4a4b-94b6-b0e7375304ed", "metadata": {}, "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "e543695a5d1b4fd691aea76872e53aaf", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "model.safetensors: 0%| | 0.00/436M [00:00\n", " \n", " \n", - " [189/189 05:39, Epoch 3/3]\n", + " [189/189 05:50, Epoch 3/3]\n", " \n", " \n", " \n", @@ -722,21 +906,21 @@ " \n", " \n", " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", " \n", " \n", " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", " \n", " \n", " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", " \n", " \n", "
11.2421001.0908860.5260001.2286001.2409870.470000
20.9014000.9601150.5910000.9117000.9681270.577000
30.6382000.9783610.5920000.6630000.9799530.583000

" @@ -751,10 +935,10 @@ { "data": { "text/plain": [ - "TrainOutput(global_step=189, training_loss=0.9693943861300353, metrics={'train_runtime': 341.7098, 'train_samples_per_second': 8.779, 'train_steps_per_second': 0.553, 'total_flos': 789354427392000.0, 'train_loss': 0.9693943861300353, 'epoch': 3.0})" + "TrainOutput(global_step=189, training_loss=1.0004738197124825, metrics={'train_runtime': 351.8582, 'train_samples_per_second': 8.526, 'train_steps_per_second': 0.537, 'total_flos': 789354427392000.0, 'train_loss': 1.0004738197124825, 'epoch': 3.0})" ] }, - "execution_count": 17, + "execution_count": 24, "metadata": {}, "output_type": "execute_result" } @@ -765,7 +949,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 25, "id": "6d581099-37a4-4470-b051-1ada38554089", "metadata": {}, "outputs": [], @@ -775,7 +959,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 26, "id": "ffb47eab-1370-491e-8a84-6d5347a350b2", "metadata": {}, "outputs": [ @@ -800,15 +984,15 @@ { "data": { "text/plain": [ - "{'eval_loss': 1.0753791332244873,\n", - " 'eval_accuracy': 0.52,\n", - " 'eval_runtime': 2.9889,\n", - " 'eval_samples_per_second': 33.457,\n", - " 'eval_steps_per_second': 4.349,\n", + "{'eval_loss': 1.111964464187622,\n", + " 'eval_accuracy': 0.53,\n", + " 'eval_runtime': 2.9464,\n", + " 'eval_samples_per_second': 33.94,\n", + " 'eval_steps_per_second': 4.412,\n", " 'epoch': 3.0}" ] }, - "execution_count": 19, + "execution_count": 26, "metadata": {}, "output_type": "execute_result" } @@ -830,7 +1014,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 27, "id": "ad0cbc14-9ef7-450f-a1a3-4f92b6486f41", "metadata": {}, "outputs": [], @@ -889,6 +1073,662 @@ "metadata": {}, "outputs": [], "source": [] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "2e31a583-9106-41bc-964e-b737c109f7fb", + "metadata": {}, + "outputs": [], + "source": [ + "from datasets import load_dataset\n", + "\n", + "dataset = load_dataset(\"yelp_review_full\")" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "575328cc-abe1-4a71-8e47-c02ace2eae18", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "DatasetDict({\n", + " train: Dataset({\n", + " features: ['label', 'text'],\n", + " num_rows: 650000\n", + " })\n", + " test: Dataset({\n", + " features: ['label', 'text'],\n", + " num_rows: 50000\n", + " })\n", + "})" + ] + }, + "execution_count": 33, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dataset" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "b562a893-40d6-48cc-8730-f15843984521", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'label': 2,\n", + " 'text': \"As far as Starbucks go, this is a pretty nice one. The baristas are friendly and while I was here, a lot of regulars must have come in, because they bantered away with almost everyone. The bathroom was clean and well maintained and the trash wasn't overflowing in the canisters around the store. The pastries looked fresh, but I didn't partake. The noise level was also at a nice working level - not too loud, music just barely audible.\\\\n\\\\nI do wish there was more seating. It is nice that this location has a counter at the end of the bar for sole workers, but it doesn't replace more tables. I'm sure this isn't as much of a problem in the summer when there's the space outside.\\\\n\\\\nThere was a treat receipt promo going on, but the barista didn't tell me about it, which I found odd. Usually when they have promos like that going on, they ask everyone if they want their receipt to come back later in the day to claim whatever the offer is. Today it was one of their new pastries for $1, I know in the summer they do $2 grande iced drinks with that morning's receipt.\\\\n\\\\nOverall, nice working or socializing environment. Very friendly and inviting. It's what I've come to expect from Starbucks, so points for consistency.\"}" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dataset[\"train\"][111]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "af82b7ed-d1d2-450f-8a37-7b9e93c7f513", + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "6711f2b4-ac67-4c4c-8fc1-93f7ded15390", + "metadata": {}, + "outputs": [], + "source": [ + "import random\n", + "import pandas as pd\n", + "import datasets\n", + "from IPython.display import display, HTML" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "4cd2f99a-84f9-4727-b6e0-78c9f0ca71bd", + "metadata": {}, + "outputs": [], + "source": [ + "def show_random_elements(dataset, num_examples=10):\n", + " assert num_examples <= len(dataset), \"Can't pick more elements than there are in the dataset.\"\n", + " picks = []\n", + " for _ in range(num_examples):\n", + " pick = random.randint(0, len(dataset)-1)\n", + " while pick in picks:\n", + " pick = random.randint(0, len(dataset)-1)\n", + " picks.append(pick)\n", + " \n", + " df = pd.DataFrame(dataset[picks])\n", + " for column, typ in dataset.features.items():\n", + " if isinstance(typ, datasets.ClassLabel):\n", + " df[column] = df[column].transform(lambda i: typ.names[i])\n", + " display(HTML(df.to_html()))" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "71e15fbf-37bb-44a8-b554-657b54caf3d0", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
labeltext
01 starThe waitress was awesome however the cook had messed up an order numerous times for a turkery burger with no cheese. They had brought out a burger with cheese, requested to put the burger on a plate. When they brought it out for a third time, it was the same burger, there was edges of the cheese still left on the meat, they had just scrapped the cheese off. When asked to speak to the cook/owner they refused to come out to speak to us. They didn't charge for the meal, however if someone was seriously allergic to cheese, this could have been a disaster. Will definitely not go back to this restaurant for sure.
11 starIt's hard to imagine only 1 star due to this hotel being above average in cost. \\nWhen I'm paying for a nicer hotel, I expect at least B+ service. We left with the impression that this hotel is all about squeezing every dollar out of you and they're not worried about feedback since there are 1500 more people checking in every day. \\n\\n1. Lazy river. They advertise this big time but they don't tell you they charge $24 for an inner tube. If not free, they should at least rent them for a few bucks, right? TIP: Go to CVS on the strip and get the same inner tube for $4. \\n\\n2. They advertise the shark reef but it costs $18 for this mildly entertaining adventure.\\n\\n3. No coffee in the rooms. Ok, my math says they save around $1,000,000 per year by not having coffee in 4000 rooms. Fine. At least have enough coffee shops in the hotel. There's only Starbucks and one other coffee place. I don't even mind the extra markup. However I waited 30 - 45 minutes in the morning for a cup of coffee with a bunch of angry people. They're also grumbling about the lazy river while waiting for coffee. \\n\\n4. After driving hours to get there, we had to wait 45 minutes to valet our car. The valet parker told me it's like that all summer long. Translation: They're too cheap to hire 2 more valet guys to make it quicker. When I left and came back that night, another 30 minutes in the valet line. Outrageous.\\n\\n5. Problems with our room: Bathtub was dirty and 2 lights didn't work. \\n\\nAny time you try to give feedback they say the same thing: Sorry, we have 1500 people checking in today.\\n\\nOn day two, I phoned guest services to tell them I'm not happy and I may check out a day early. Their response: \\\"Ok sir, I have you leaving on the 27th now. Any thing else?\\\" They didn't even ask what the problem was. \\n\\nI filled out an email survey stating all of the above and asked them to contact me. NO RESPONSE. My wife is so easy to please and she encouraged me to check out early. We came across dozens of guests that were having a similar experience. Keep ignoring us but eventually it will catch up with this hotel. \\n\\nI will never stay here again and it's a shame. If they just cared a little, it would be a great place. The wave pool and lazy river are fun for kids.
21 starThe service was terrible. I would go back again because the food was good but would not go back just because of the service.
35 starsWhat an awesome way to start the night. Just before going to work i came into just for appetizers and a few cocktails. The staff was great, the ambiance was phenominal and the drink prices were decent. i defiantly found a good place to bring dates or even a friend visiting. plus Light Nightclub is next door !!!!
44 starsThis location has gotten much better lately. They have the best balloon guy here. He does great balloon- items. Kids got a motorcycle and a helicopter today. The service has gotten much better. They are actually starting to remember to bring silverware instead of making the customers ask for it. The only thing that is better at other locations is that the rolls are usually overdone at this location, which is why I only gave 4 stars. Keep up the good work, Gilbert and Germann location. Just take the bread out a bit earlier so it doesn't overcook and this location would be a 5-star location
53 starsThe spicy tofu soup was okay. The sides were great. The fresh noodle soup was bland. Service was nice.
62 starPho here was ok, broth was to salty for my taste even after squeezing in half a lemon. Portions were decent, the server was very nice and pleasant. The middle asian man taking our order was kinda sad looking... Didn't even crack a smile..
74 starsThe best Philly I've ever had and I am not a huge red meat fan. The sandwiches are huge my 2 teenage boys have a hard time finishing these so who cares if they're a bit pricey.\\nDang now I'm craving one!
82 starThis place is another Vegas night club that is living off its reputation. \\n\\nFirst, the dance floor is way too crowded for any real dancing. I could barely move without bumping into other people. Second, my girlfriend and I waited in line way too long while some average looking girls at best cut to the front of the line. If you are looking for a place to dance and have a few drinks without a bunch of hassles don't come here.
92 star20 years I waited for this slop. Everything was lukewarm at best. Desserts that were supposed to be cold were warm. My 17 dollar cocktail gave me a good buzz for 5 minutes so 1 star for that. 2nd star is for service. Remember people...its a buffet. Lower your expectations or you will be slapped in the face by disappointment like myself.
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "show_random_elements(dataset[\"train\"])" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "id": "b5903810-ff61-4008-8850-6df5e4ef7c74", + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import AutoTokenizer\n", + "\n", + "tokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")\n", + "\n", + "\n", + "def tokenize_function(examples):\n", + " return tokenizer(examples[\"text\"], padding=\"max_length\", truncation=True)\n", + "\n", + "\n", + "tokenized_datasets = dataset.map(tokenize_function, batched=True)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "0539f66a-a0a8-429b-b2a6-f76cd2dc9c16", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForSequenceClassification: ['cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias']\n", + "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n", + "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n", + "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\n", + "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n" + ] + } + ], + "source": [ + "from transformers import AutoModelForSequenceClassification\n", + "\n", + "model = AutoModelForSequenceClassification.from_pretrained(\"bert-base-cased\", num_labels=5)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "16a4afa2-1eb8-4a0a-92c5-c187b59b0585", + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "5aba8525-705e-423d-82e2-ebca020dde0b", + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import TrainingArguments, Trainer\n", + "model_dir = \"models/bert-base-cased-finetune-yelp\"\n", + "training_args = TrainingArguments(output_dir=model_dir,\n", + " evaluation_strategy=\"epoch\", \n", + " save_strategy=\"steps\", # 设置保存策略为按步骤保存\n", + " save_steps=500, # 每500步保存一次检查点\n", + " save_total_limit=5, # 最多保存5个检查点，超过限制时会删除旧的检查点\n", + " per_device_train_batch_size=16,\n", + " num_train_epochs=3,\n", + " logging_steps=30)" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "27380b1a-379f-461f-ac1f-3f94f2d02457", + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "import evaluate\n", + "\n", + "metric = evaluate.load(\"accuracy\")\n", + "\n", + "def compute_metrics(eval_pred):\n", + " logits, labels = eval_pred\n", + " predictions = np.argmax(logits, axis=-1)\n", + " return metric.compute(predictions=predictions, references=labels)" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "173083a3-2e6b-46e5-b69b-13536ea58d57", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n" + ] + } + ], + "source": [ + "train_dataset = tokenized_datasets[\"train\"]\n", + "eval_dataset = tokenized_datasets[\"test\"]\n", + "trainer = Trainer(\n", + " model=model,\n", + " args=training_args,\n", + " train_dataset=train_dataset,\n", + " eval_dataset=eval_dataset,\n", + " compute_metrics=compute_metrics,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "64a1252c-43d1-4d40-8482-d75df02d00a9", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/root/miniconda3/envs/myenv/lib/python3.11/site-packages/transformers/optimization.py:411: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n", + " warnings.warn(\n" + ] + }, + { + "data": { + "text/html": [ + "\n", + "

\n", + " \n", + " \n", + " [ 17658/121875 6:56:34 < 40:58:55, 0.71 it/s, Epoch 0.43/3]\n", + "
\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
EpochTraining LossValidation Loss

" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "IOPub message rate exceeded.\n", + "The Jupyter server will temporarily stop sending output\n", + "to the client in order to avoid crashing it.\n", + "To change this limit, set the config variable\n", + "`--ServerApp.iopub_msg_rate_limit`.\n", + "\n", + "Current values:\n", + "ServerApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", + "ServerApp.rate_limit_window=3.0 (secs)\n", + "\n" + ] + } + ], + "source": [ + "trainer.train()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5a9c26c7-c086-423c-8442-a17ef53242d1", + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "5f1d06af-6e6d-4ce4-ba05-7ac5e97cf540", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "

\n", + " \n", + " \n", + " [13/13 00:02]\n", + "
\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/plain": [ + "{'eval_loss': 1.111964464187622,\n", + " 'eval_accuracy': 0.53,\n", + " 'eval_runtime': 2.9464,\n", + " 'eval_samples_per_second': 33.94,\n", + " 'eval_steps_per_second': 4.412,\n", + " 'epoch': 3.0}" + ] + }, + "execution_count": 26, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "trainer.evaluate(test_dataset)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "86bf03d1-dd50-42ca-b1e3-8321dea6bebc", + "metadata": {}, + "outputs": [], + "source": [ + "#重新开始训练：" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "1e69a149-ddb6-4a7d-88b5-41e733f2bc1a", + "metadata": {}, + "outputs": [], + "source": [ + "model_dir = \"models/bert-base-cased-finetune-yelp\"\n", + "trainer.save_model(model_dir)" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "ca8fcc64-4680-4edb-abac-deed2e8ed5cb", + "metadata": {}, + "outputs": [], + "source": [ + "trainer.save_state()" + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "id": "80af1bde-21a4-4dbc-8491-01b24aaf1ca5", + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import AutoModelForSequenceClassification\n", + "\n", + "model = AutoModelForSequenceClassification.from_pretrained(\"models/bert-base-cased-finetune-yelp/checkpoint-40500\", num_labels=5)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1c580c89-5071-41c4-9270-a5b6966ae13d", + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": 39, + "id": "c37156ee-fbf1-4a0b-805c-1af7fd1a3b98", + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import TrainingArguments, Trainer\n", + "model_dir = \"models/bert-base-cased-finetune-yelp\"\n", + "training_args = TrainingArguments(output_dir=model_dir,\n", + " evaluation_strategy=\"epoch\", \n", + " save_strategy=\"steps\", # 设置保存策略为按步骤保存\n", + " save_steps=500, # 每500步保存一次检查点\n", + " save_total_limit=5, # 最多保存5个检查点，超过限制时会删除旧的检查点\n", + " per_device_train_batch_size=16,\n", + " num_train_epochs=3,\n", + " logging_steps=30)" + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "id": "715c7d19-1635-499c-bbb5-de76bba59559", + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "import evaluate\n", + "\n", + "metric = evaluate.load(\"accuracy\")\n", + "\n", + "def compute_metrics(eval_pred):\n", + " logits, labels = eval_pred\n", + " predictions = np.argmax(logits, axis=-1)\n", + " return metric.compute(predictions=predictions, references=labels)" + ] + }, + { + "cell_type": "code", + "execution_count": 43, + "id": "2f5da60e-5d3f-49cc-a95f-5133dac2dac6", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n" + ] + } + ], + "source": [ + "train_dataset = tokenized_datasets[\"train\"]\n", + "eval_dataset = tokenized_datasets[\"test\"]\n", + "trainer = Trainer(\n", + " model=model,\n", + " args=training_args,\n", + " train_dataset=train_dataset,\n", + " eval_dataset=eval_dataset,\n", + " compute_metrics=compute_metrics,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7b1faf6c-057f-4305-9098-345af1e32845", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " [ 63879/121875 9:32:25 < 23:40:08, 0.68 it/s, Epoch 1.57/3]\n", + "
\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
EpochTraining LossValidation LossAccuracy
10.7236000.7729140.669640

" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "trainer.train(resume_from_checkpoint=\"models/bert-base-cased-finetune-yelp/checkpoint-40500\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dce6e032-32d4-4a48-ade9-962f3fba7dd0", + "metadata": {}, + "outputs": [], + "source": [ + "notebook出现了不同步的情况，从保存的check point来看，训练完成了，现在以保存的最后一个checkpoint开始重新训练：" + ] + }, + { + "cell_type": "code", + "execution_count": 47, + "id": "07badef1-d292-4857-9d5e-eb25e77cf2c2", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "

\n", + " \n", + " \n", + " [121875/121875 34:27, Epoch 3/3]\n", + "
\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
EpochTraining LossValidation LossAccuracy
30.4874000.7377630.688320

" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/plain": [ + "TrainOutput(global_step=121875, training_loss=0.001603093993358123, metrics={'train_runtime': 2068.9667, 'train_samples_per_second': 942.499, 'train_steps_per_second': 58.906, 'total_flos': 5.1308879758535885e+17, 'train_loss': 0.001603093993358123, 'epoch': 3.0})" + ] + }, + "execution_count": 47, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "trainer.train(resume_from_checkpoint=\"models/bert-base-cased-finetune-yelp/checkpoint-121500\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "685636be-2e2c-4906-b307-ddcf83b7f8df", + "metadata": {}, + "outputs": [], + "source": [ + "结论： 训练完整的训练集，Acc提高到0.688320" + ] } ], "metadata": { diff --git a/transformers/pipelines.ipynb b/transformers/pipelines.ipynb index a2d26e3..95f0e09 100644 --- a/transformers/pipelines.ipynb +++ b/transformers/pipelines.ipynb @@ -657,20 +657,62 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 1, "id": "2e1f8a8d-75eb-49ab-9353-6c9d1f384cac", "metadata": {}, "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "2f17930874264274836012321f367177", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "config.json: 0%| | 0.00/1.66k [00:00=1.7 in /root/miniconda3/lib/python3.11/site-packages (from timm) (2.3.0.dev20240116+cu121)\n", - "Requirement already satisfied: torchvision in /root/miniconda3/lib/python3.11/site-packages (from timm) (0.18.0.dev20240117+cu121)\n", - "Requirement already satisfied: pyyaml in /root/miniconda3/lib/python3.11/site-packages (from timm) (6.0.1)\n", - "Requirement already satisfied: huggingface-hub in /root/miniconda3/lib/python3.11/site-packages (from timm) (0.20.1)\n", - "Requirement already satisfied: safetensors in /root/miniconda3/lib/python3.11/site-packages (from timm) (0.4.0)\n", - "Requirement already satisfied: filelock in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (3.9.0)\n", - "Requirement already satisfied: typing-extensions>=4.8.0 in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (4.8.0)\n", - "Requirement already satisfied: sympy in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (1.11.1)\n", - "Requirement already satisfied: networkx in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (3.0rc1)\n", - "Requirement already satisfied: jinja2 in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (3.1.2)\n", - "Requirement already satisfied: fsspec in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (2023.4.0)\n", - "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (12.1.105)\n", - "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (12.1.105)\n", - "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (12.1.105)\n", - "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (8.9.2.26)\n", - "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (12.1.3.1)\n", - "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (11.0.2.54)\n", - "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (10.3.2.106)\n", - "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (11.4.5.107)\n", - "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (12.1.0.106)\n", - "Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (2.19.3)\n", - "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (12.1.105)\n", - "Requirement already satisfied: pytorch-triton==2.2.0+e28a256d71 in /root/miniconda3/lib/python3.11/site-packages (from torch>=1.7->timm) (2.2.0+e28a256d71)\n", - "Requirement already satisfied: nvidia-nvjitlink-cu12 in /root/miniconda3/lib/python3.11/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.7->timm) (12.1.105)\n", - "Collecting fsspec (from torch>=1.7->timm)\n", - " Using cached fsspec-2023.12.2-py3-none-any.whl.metadata (6.8 kB)\n", - "Requirement already satisfied: requests in /root/miniconda3/lib/python3.11/site-packages (from huggingface-hub->timm) (2.28.1)\n", - "Requirement already satisfied: tqdm>=4.42.1 in /root/miniconda3/lib/python3.11/site-packages (from huggingface-hub->timm) (4.66.1)\n", - "Requirement already satisfied: packaging>=20.9 in /root/miniconda3/lib/python3.11/site-packages (from huggingface-hub->timm) (23.2)\n", - "Requirement already satisfied: numpy in /root/miniconda3/lib/python3.11/site-packages (from torchvision->timm) (1.24.1)\n", - "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /root/miniconda3/lib/python3.11/site-packages (from torchvision->timm) (9.3.0)\n", - "Requirement already satisfied: MarkupSafe>=2.0 in /root/miniconda3/lib/python3.11/site-packages (from jinja2->torch>=1.7->timm) (2.1.3)\n", - "Requirement already satisfied: charset-normalizer<3,>=2 in /root/miniconda3/lib/python3.11/site-packages (from requests->huggingface-hub->timm) (2.1.1)\n", - "Requirement already satisfied: idna<4,>=2.5 in /root/miniconda3/lib/python3.11/site-packages (from requests->huggingface-hub->timm) (3.4)\n", - "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /root/miniconda3/lib/python3.11/site-packages (from requests->huggingface-hub->timm) (1.26.13)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /root/miniconda3/lib/python3.11/site-packages (from requests->huggingface-hub->timm) (2022.12.7)\n", - "Requirement already satisfied: mpmath>=0.19 in /root/miniconda3/lib/python3.11/site-packages (from sympy->torch>=1.7->timm) (1.2.1)\n", - "Using cached fsspec-2023.12.2-py3-none-any.whl (168 kB)\n", - "Installing collected packages: fsspec\n", - " Attempting uninstall: fsspec\n", - " Found existing installation: fsspec 2023.4.0\n", - " Uninstalling fsspec-2023.4.0:\n", - " Successfully uninstalled fsspec-2023.4.0\n", - "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n", - "datasets 2.16.1 requires fsspec[http]<=2023.10.0,>=2023.1.0, but you have fsspec 2023.12.2 which is incompatible.\u001b[0m\u001b[31m\n", - "\u001b[0mSuccessfully installed fsspec-2023.12.2\n", + "Requirement already satisfied: timm in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (0.9.12)\n", + "Requirement already satisfied: torch>=1.7 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from timm) (2.2.0)\n", + "Requirement already satisfied: torchvision in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from timm) (0.17.0)\n", + "Requirement already satisfied: pyyaml in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from timm) (6.0.1)\n", + "Requirement already satisfied: huggingface-hub in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from timm) (0.20.2)\n", + "Requirement already satisfied: safetensors in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from timm) (0.4.2)\n", + "Requirement already satisfied: filelock in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (3.13.1)\n", + "Requirement already satisfied: typing-extensions>=4.8.0 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (4.9.0)\n", + "Requirement already satisfied: sympy in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (1.12)\n", + "Requirement already satisfied: networkx in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (3.2.1)\n", + "Requirement already satisfied: jinja2 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (3.1.3)\n", + "Requirement already satisfied: fsspec in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (2023.10.0)\n", + "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (12.1.105)\n", + "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (12.1.105)\n", + "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (12.1.105)\n", + "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (8.9.2.26)\n", + "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (12.1.3.1)\n", + "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (11.0.2.54)\n", + "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (10.3.2.106)\n", + "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (11.4.5.107)\n", + "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (12.1.0.106)\n", + "Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (2.19.3)\n", + "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (12.1.105)\n", + "Requirement already satisfied: triton==2.2.0 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torch>=1.7->timm) (2.2.0)\n", + "Requirement already satisfied: nvidia-nvjitlink-cu12 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.7->timm) (12.3.101)\n", + "Requirement already satisfied: requests in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from huggingface-hub->timm) (2.31.0)\n", + "Requirement already satisfied: tqdm>=4.42.1 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from huggingface-hub->timm) (4.65.0)\n", + "Requirement already satisfied: packaging>=20.9 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from huggingface-hub->timm) (23.2)\n", + "Requirement already satisfied: numpy in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torchvision->timm) (1.26.3)\n", + "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from torchvision->timm) (10.2.0)\n", + "Requirement already satisfied: MarkupSafe>=2.0 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from jinja2->torch>=1.7->timm) (2.1.5)\n", + "Requirement already satisfied: charset-normalizer<4,>=2 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from requests->huggingface-hub->timm) (3.3.2)\n", + "Requirement already satisfied: idna<4,>=2.5 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from requests->huggingface-hub->timm) (3.6)\n", + "Requirement already satisfied: urllib3<3,>=1.21.1 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from requests->huggingface-hub->timm) (2.2.0)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from requests->huggingface-hub->timm) (2024.2.2)\n", + "Requirement already satisfied: mpmath>=0.19 in /root/miniconda3/envs/myenv/lib/python3.11/site-packages (from sympy->torch>=1.7->timm) (1.3.0)\n", "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n", "\u001b[0m" ] @@ -1054,7 +1263,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 11, "id": "c3401126-756b-4394-930c-c833c1f574b0", "metadata": {}, "outputs": [ @@ -1063,13 +1272,79 @@ "output_type": "stream", "text": [ "No model was supplied, defaulted to facebook/detr-resnet-50 and revision 2729413 (https://huggingface.co/facebook/detr-resnet-50).\n", - "Using a pipeline without specifying a model name and revision in production is not recommended.\n", - "/root/miniconda3/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libpng16.so.16: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?\n", - " warn(\n", - "Some weights of the model checkpoint at facebook/detr-resnet-50 were not used when initializing DetrForObjectDetection: ['model.backbone.conv_encoder.model.layer2.0.downsample.1.num_batches_tracked', 'model.backbone.conv_encoder.model.layer3.0.downsample.1.num_batches_tracked', 'model.backbone.conv_encoder.model.layer4.0.downsample.1.num_batches_tracked', 'model.backbone.conv_encoder.model.layer1.0.downsample.1.num_batches_tracked']\n", + "Using a pipeline without specifying a model name and revision in production is not recommended.\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "4a650f3c66c84662a6bda6edc4df6ff4", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "config.json: 0%| | 0.00/4.59k [00:00