{"id":9169,"date":"2023-03-04T16:01:05","date_gmt":"2023-03-04T16:01:05","guid":{"rendered":"https:\/\/froginawell.net\/frog\/?p=9169"},"modified":"2023-03-04T17:27:55","modified_gmt":"2023-03-04T17:27:55","slug":"cleaning-up-tables-from-primary-sources-in-chatgpt","status":"publish","type":"post","link":"https:\/\/froginawell.net\/frog\/2023\/03\/cleaning-up-tables-from-primary-sources-in-chatgpt\/","title":{"rendered":"Cleaning Up Tables from Primary Sources in ChatGPT"},"content":{"rendered":"

I’ve been following with interest the debates around the rapid emergence of powerful large language models such as OpenAI’s ChatGPT, its Bing sibling Sydney, Meta’s Galactica, and Google’s Bard. One important recent discussion of this can be found here<\/a>. My current status: deep concern mixed with pragmatic curiosity.<\/p>\n

\"Summation<\/a><\/p>\n

Given the propensity of ChatGPT (mid-February, 2023 version) to happily invent facts, people, nonexistant citations, and quotations, I’m not yet too worried about how this impacts historical essays produced by students. However, while its shortcomings in this regard may give only temporary relief as these models evolve, it also limits its usefulness for quick information lookups on things you are not already expert enough on to call bullshit on. So are there any current use cases for historians? I stumbled on one potential use through a post<\/a> on Mastodon: apparently, ChatGPT is not bad at cleaning up and formatting tables from raw text.<\/p>\n

 <\/p>\n

To test this, I took some very badly formatted data from a single table randomly chosen from my photo of a March, 1946 issue of a Summation of United States Army Military Government Activities in Korea. <\/em>Here is a view of the original table:<\/p>\n

\"\"<\/a><\/p>\n

Here is the poorly formatted text extracted from this:<\/p>\n

From   To    Quantity (in suk)\nCholla Pukto  Seoul  32,000\n             Inchon       6,000\n                           Mukko       8,000\n                       Chechon  2,000\n                       Wonju  2,000\n                       Chunchon    8,000\n                __________\n                    58,000\nCholla Namdo    Mukko              8,000\nChung Chong Pukto       Seoul 4,500\nChung Chong Namdo   Seoul 36,000\n                Inchon          6,000\n                    ______\n                         42,000\nKyong Sang Pukto   Chechon       5,000\n                 Mukko    7,0000\n                    _______________\n                    12,000\n\n<\/pre>\n
Now give ChatGPT the instructions to clean up the table:<\/p>\n
\"\"<\/a><\/p>\n
This produced the following:<\/p>\n
\"\"<\/a><\/p>\n
Selecting the text from the entire table, copying (from Firefox browser, in my case) and pasting it directly into (Mac) TextEdit or Microsoft Word retained the table formatting perfectly. But here it is again below. I asked ChatGPT to reproduce the table as HTML for easy copy\/paste here with prompt: “reproduce this table as html”:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
From<\/th>\nTo<\/th>\nQuantity (in suk)<\/th>\n<\/tr>\n
Cholla Pukto<\/td>\nSeoul<\/td>\n32,000<\/td>\n<\/tr>\n
<\/td>\nInchon<\/td>\n6,000<\/td>\n<\/tr>\n
<\/td>\nMukko<\/td>\n8,000<\/td>\n<\/tr>\n
<\/td>\nChechon<\/td>\n2,000<\/td>\n<\/tr>\n
<\/td>\nWonju<\/td>\n2,000<\/td>\n<\/tr>\n
<\/td>\nChunchon<\/td>\n8,000<\/td>\n<\/tr>\n
Total<\/strong><\/td>\n58,000<\/strong><\/td>\n<\/tr>\n
Cholla Namdo<\/td>\nMukko<\/td>\n8,000<\/td>\n<\/tr>\n
Chung Chong Pukto<\/td>\nSeoul<\/td>\n4,500<\/td>\n<\/tr>\n
Chung Chong Namdo<\/td>\nSeoul<\/td>\n36,000<\/td>\n<\/tr>\n
<\/td>\nInchon<\/td>\n6,000<\/td>\n<\/tr>\n
Total<\/strong><\/td>\n42,000<\/strong><\/td>\n<\/tr>\n
Kyong Sang Pukto<\/td>\nChechon<\/td>\n5,000<\/td>\n<\/tr>\n
<\/td>\nMukko<\/td>\n7,0000<\/td>\n<\/tr>\n
Total<\/strong><\/td>\n12,000<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
You might notice that the model added “total” where it detected sub-totals for some sections. Also notice that the mistakenly transcribed 7,0000 is converted to 7,000 in the reply, but back to 7,0000 in the HTML table when I requested it in the next prompt (h\/t to Janis<\/a> for noticing this). Clearly a reminder to check the results as carefully as with OCR outputs.<\/p>\n
There are lots of other places online that offer services for cleaning up messy data, but I have had mixed results with them. This worked quite well and can potentially save a lot of time cleaning up tabular data in OCRs of historical documents.<\/p>\n","protected":false},"excerpt":{"rendered":"
I’ve been following with interest the debates around the rapid emergence of powerful large language models such as OpenAI’s ChatGPT, its Bing sibling Sydney, Meta’s Galactica, and Google’s Bard. One important recent discussion of this can be found here. My current status: deep concern mixed with pragmatic curiosity. Given the propensity of ChatGPT (mid-February, 2023…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[280],"tags":[],"class_list":["post-9169","post","type-post","status-publish","format-standard","hentry","category-posts"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p9yoH3-2nT","_links":{"self":[{"href":"https:\/\/froginawell.net\/frog\/wp-json\/wp\/v2\/posts\/9169","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/froginawell.net\/frog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/froginawell.net\/frog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/froginawell.net\/frog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/froginawell.net\/frog\/wp-json\/wp\/v2\/comments?post=9169"}],"version-history":[{"count":8,"href":"https:\/\/froginawell.net\/frog\/wp-json\/wp\/v2\/posts\/9169\/revisions"}],"predecessor-version":[{"id":9184,"href":"https:\/\/froginawell.net\/frog\/wp-json\/wp\/v2\/posts\/9169\/revisions\/9184"}],"wp:attachment":[{"href":"https:\/\/froginawell.net\/frog\/wp-json\/wp\/v2\/media?parent=9169"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/froginawell.net\/frog\/wp-json\/wp\/v2\/categories?post=9169"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/froginawell.net\/frog\/wp-json\/wp\/v2\/tags?post=9169"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}